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EXECUTIVE  SUMMARY 


The  objective  of  this  ESTCP  project  was  to  demonstrate  and  validate  the  use  of  the  “Sampling 
Optimizer  and  Data  Tracker”  software  (the  “Summit  Software”),  offered  by  Summit 
Envirosolutions,  at  three  DoD  sites.  The  three  demonstration  sites  were  as  follows: 

•  Eormer  George  Air  Eorce  Base  Site,  Victorville,  CA  (GAEB  site) 

•  Eormer  Nebraska  Ordnance  Plant  Site,  Mead,  NE  (NOP  site) 

•  Camp  Allen  Eandfill  Site,  Norfolk,  VA  (Camp  Allen  site) 

MAROS  (developed  by  GSI  Environmental  Inc.)  was  also  applied  at  one  of  the  three 
demonstration  sites.  The  Summit  Software  demonstrated  in  this  ESTCP  project  provides  a  set  of 
tools  for  Eong-Term  Monitoring  Optimization  (ETMO),  and  consists  of  two  major  modules: 

•  Sampling  Optimizer  (SO)  identifies  redundant  sampling  locations  (spatial  optimization), 
or  redundant  locations  and  frequencies  (spatiotemporal  optimization),  in  historical  data. 

•  Data  Tracker  (DT)  allows  current  monitoring  data  to  be  reviewed  against  selected 
historical  data  (i.e.,  the  “background  data”)  to  identify  cases  where  current  data  deviate 
from  expectations  that  are  based  on  the  background  values  and  patterns. 

Model  Builder  is  an  additional  component  within  the  software  with  two  functions:  one  for  model 
fitting,  visualization,  and  analysis  (with  kriging  or  inverse  distance  weighting);  and  another  for 
visualizing  relative  uncertainty. 

Key  results  of  the  project  include  the  following: 

•  The  software  is  easy  to  learn/use,  and  no  bugs  or  software  errors  are  apparent. 

•  Of  the  six  options  available,  kriging  with  quantile  data  transformation  qualitatively 
provided  the  best  representation  of  the  plumes  with  Model  Builder. 

•  Sampling  Optimizer  provided  useful  trade-off  curves  of  sampling  cost  versus  the 
interpolation  error  that  resulted  from  removing  samples. 

•  Potential  savings  from  spatial  analysis  (i.e.,  eliminating  redundant  sampling  locations), 
calculated  based  on  number  of  wells  eliminated,  ranged  from  approximately  10%  to 
approximately  67%.  A  value  of  approximately  35%  appears  to  be  representative.  Eor  the 
two  sites  where  spatiotemporal  analysis  was  performed,  the  potential  savings,  based  on 
number  of  samples  eliminated  per  year,  only  ranged  from  approximately  4%  to 
approximately  17%.  Reasons  why  overly  conservative  results  may  be  obtained  from  the 
spatiotemporal  analysis  (versus  spatial  analysis)  are  discussed  in  the  Technical  Report. 
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•  The  software  allows  redundaney  analysis  to  be  performed  simultaneously  for  multiple 
eontaminants  of  eoneern.  Based  on  the  testing  that  was  performed  it  appears  that 
simultaneous  evaluation  is  preferable  to  evaluating  multiple  eonstituents  independently. 

•  Validation  exereises  eondueted  with  reserved  data  provided  eonfidenee  in  the  results 
provided  by  Sampling  Optimizer  at  all  three  demonstration  sites. 

•  If  distribution  of  sampling  loeations  is  not  eonsistent  between  events,  the  results  of 
relative  mass  or  mass  flux  caleulations  provided  by  the  software  may  be  questionable,  for 
reasons  discussed  in  the  Technical  Report.  This  limitation  also  exists  for  MAROS. 

•  The  relative  uncertainty  maps  provided  by  Model  Builder  were  not  found  to  be 
particularly  useful. 

•  Data  Tracker  identified  as  “out-of-bounds”  the  vast  majority  of  artificial  anomalies  added 
by  EnviroStat  for  testing  of  this  module,  and  also  identified  some  anomalies  in  the  actual 
site  data. 

Some  limitations  of  the  software  were  identified  that  could  potentially  be  mitigated  by  future 
software  improvements,  including  the  following: 

•  The  software  interpolates  spatially  but  does  not  perform  interpolations  in  time.  The 
software  would  be  improved  if  there  was  a  feature  to  optionally  fill  in  missing  values  via 
temporal  interpolation. 

•  In  DT,  the  software  would  be  improved  if  plots  of  concentration  versus  time  used 
different  symbols  to  differentiate  between  the  “background  data”  and  the  “current  data”. 

•  In  DT,  the  software  would  be  improved  if  selected  values  could  be  imported  with  a  flag 
so  that  they  can  be  included  on  concentration  versus  time  plots  (with  a  different  symbol) 
but  not  used  to  calculate  the  prediction  limits. 

•  The  DT  portion  of  the  software  would  be  improved  if  it  also  indicated  whether  the 
concentration  trend  for  a  specific  COC  at  a  specific  well  is  increasing,  stable,  or 
decreasing  (i.e.,  as  is  provided  by  MAROS). 

•  The  software  import  features  would  be  improved  by  adding  features  to  consolidate  data 
into  discrete  events  and  to  recognize  flags  (e.g.,  for  non-detect  values). 

The  level  of  effort  and  computation  time  for  applying  the  software  at  the  demonstration  sites,  and 
a  basis  for  estimating  the  costs  of  applying  the  software  at  other  sites,  are  provided  in  this  report. 
The  software  and  User’s  Guide  are  now  available  for  use  at  government  sites  by  government 
personnel  and  their  contractors,  which  was  an  additional  component  of  this  ESCTP  project. 
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1.0  INTRODUCTION 


1.1  BACKGROUND 

Long-term  monitoring  (LTM)  provides  a  meehanism  for  evaluating  performanee  of  groundwater 
remedies  and  is  essential  for  ensuring  proteetion  of  human  health  and  the  environment.  The 
eosts  of  future  monitoring  are  expected  to  be  substantial,  since  LTM  generally  spans  many  years 
and  is  required  at  a  large  number  of  sites.  Efficiency  of  LTM  can  be  improved  by  the  following: 

•  obtaining  only  the  essential  data  needed  for  monitoring  current  conditions  by  eliminating 
redundant  sampling  locations  and/or  frequencies; 

•  using  a  semi-automated  approach  to  identify  values  from  recently  collected  data  that  are 
not  within  expectations  (based  on  statistical  evaluation  of  previous  values);  and 

•  tracking  performance  relative  to  specific  metrics  (e.g.,  assessing  reductions  in  overall 
contaminant  mass). 

This  project  demonstrates  the  application  of  the  “Sampling  Optimizer  and  Data  Tracker” 
software  offered  by  Summit  Envirosolutions,  Inc.  (Summit),  which  is  intended  to  address  the 
items  listed  above.  We  sometimes  refer  to  this  software  as  the  “Summit  Software”. 

The  evaluation  of  data  redundancy  in  the  Sampling  Optimizer  (SO)  portion  of  the  software  uses 
mathematical  optimization,  which  is  unique  relative  to  other  ETM  optimization  (ETMO) 
software  products.  This  allows  sampling  redundancy  to  be  evaluated  on  a  system-wide  basis 
(e.g.,  best  solution  if  one  location  is  removed,  if  two  locations  are  removed,  if  three  locations  are 
removed,  etc.).  A  key  benefit  of  this  approach  is  that  it  allows  the  tradeoff  between  the  number 
of  samples  and  the  accuracy  of  the  resulting  plume  interpolation  to  be  assessed.  This  is  a 
significant  improvement  over  the  approach  for  evaluating  data  redundancy  utilized  in  the 
Monitoring  and  Remediation  Optimization  Software  (MAROS)  software  (developed  by  GSI 
Environmental  Inc.),  which  is  not  based  on  mathematical  optimization.  In  MAROS,  individual 
wells  locations  are  evaluated  for  redundancy  based  on  impacts  of  removing  that  well  alone; 
consequently,  the  impact  of  removing  groups  of  wells  cannot  be  assessed  and  the 
aforementioned  tradeoff  cannot  be  evaluated.  Another  key  benefit  of  the  Summit  approach  for 
evaluating  data  redundancy  is  that  plume  visualizations  for  the  baseline  plan  (i.e.,  all  samples) 
versus  improved  plans  (i.e.,  reduced  numbers  of  samples)  are  created  within  the  software.  These 
comparative  visualizations  are  quite  effective  for  communication  with  stakeholders  and 
regulators. 

The  “Data  Tracker”  (DT)  portion  of  the  software  identifies  values  from  recently  collected  data 
that  are  not  within  expectations  (based  on  statistical  evaluation  of  previous  values).  This  is 
somewhat  different  in  approach  and  implementation  versus  MAROS,  which  evaluates 
concentration  trends  over  time  at  individual  wells  as  increasing,  decreasing,  or  stable.  Rather 
than  indicate  increasing  versus  decreasing  trends,  the  intent  of  the  DT  portion  of  the  software  is 
to  automatically  highlight  which  recently  collected  data  values  are  unexpected  and  require 
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further  attention.  These  unexpeeted  values  may  be  due  to  signifieant  inereasing  or  deereasing 
trends,  or  may  be  due  to  “bad  data”  (lab  error,  sampling  error,  database  error,  ete.).  DT  allows 
for  “expeeted”  time  trends  to  be  either  stable  or  smoothly  deereasing;  the  latter  is  appropriate  for 
monitoring  an  effeetive  passive  remediation  system,  for  example. 


1.2  OBJECTIVE  OF  THE  PROJECT 

The  objeetive  of  this  projeet  is  to  demonstrate  and  validate  the  use  of  the  “Sampling  Optimizer 
and  Data  Traeker”  software  (i.e.,  the  “Summit  Software”)  by  applying  the  software  at  three 
Department  of  Defense  (DoD)  sites.  A  seeondary  objeetive  is  to  eompare  the  results  with 
MAROS  at  one  site.  Another  eomponent  of  the  projeet  is  to  transfer  the  software  and 
doeumentation  to  the  government  for  free  use  at  government  sites  by  government  personnel  and 
their  eontraetors. 


1.3  REGULATORY  DRIVERS 

There  are  no  regulatory  issues  direetly  assoeiated  with  this  effort,  although  there  has  been  a 
general  foeus  in  reeent  years  regarding  optimization  of  all  faeets  of  remediation  ineluding  LTM. 
Applieation  of  the  software  demonstrated  in  this  projeet  is  intended  to  improve  the  effieieney  and 
assessment  of  the  monitoring  well  networks  and  data  that  are  eolleeted  in  eurrent  monitoring 
events,  whieh  will  ultimately  address  regulatory  objeetives  and  allow  for  improved 
eommunieation  between  all  site  stakeholders.  Implementation  of  revised  sampling  plans 
suggested  for  the  demonstration  sites  is  not  within  the  seope  of  this  projeet. 
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2.0  TECHNOLOGY 


2,1  TECHNOLOGY  DESCRIPTION 

The  Summit  Software  is  a  set  of  desktop  software  tools  eonsisting  of  two  major  modules, 
Sampling  Optimizer  and  Data  Tracker. 

•  Samplins  Optimizer  (SO)  identifies  redundant  sampling  locations  (spatial  optimization), 
or  redundant  locations  and  frequencies  (spatiotemporal  optimization),  in  historical  data. 
This  module  identifies  redundancies  using  a  multi-objective  genetic  algorithm  (GA)  to 
obtain  monitoring  designs  that  represent  optimal  tradeoffs  among  two  or  more 
monitoring  objectives,  such  as  minimizing  the  number  of  samples  and  minimizing  the 
interpolation  error  at  locations  that  are  removed.  The  error  will  generally  increase  as 
number  of  wells  decreases,  resulting  in  a  tradeoff 

•  Data  Tracker  (DT)  allows  current  monitoring  data  to  be  reviewed  against  selected 
historical  data  (i.e.,  the  “background  data”)  to  identify  cases  where  current  data  deviate 
from  expectations  that  are  based  on  the  background  values  and  patterns. 

Model  Builder  is  an  additional  component  within  the  software  that  is  utilized  by  Sampling 
Optimizer  and,  in  some  cases,  by  Data  Tracker.  Model  Builder  has  two  sections:  one  for  model 
fitting,  visualization,  and  analysis  (with  kriging  or  inverse  distance  weighting);  and  another  for 
visualizing  relative  uncertainty. 

The  software  modules/components  listed  above  (Sampling  Optimizer,  Data  Tracker,  and  Model 
Builder)  are  highlighted  on  Figure  2-1,  which  is  a  general  flowchart  illustrating  the  application  of 
the  software.  The  software  is  applied  in  several  ways: 

•  During  initial  optimization  and  periodic  review  (upper  portion  of  the  figure).  Model 
Builder  constructs  spatial  and/or  spatiotemporal  models  for  the  measurements  of  the 
primary  contaminants  of  concern  (COCs).  The  model  identified  by  Model  Builder  is  then 
used  by  Sampling  Optimizer  to  identify  optimal  sampling  plans  for  subsequent  routine 
monitoring  (i.e.,  with  redundancies  eliminated). 

•  After  new  sampling  events  (lower  portion  of  the  figure)  the  software  can  be  used  during 
routine  monitoring  to  identify  anomalies  or  departures  from  expectations  in  the  primary 
COCs  and  other  COCs  as  desired.  In  addition,  changes  over  time  in  plume  mass  and/or 
mass  flux  across  a  boundary  can  be  tracked.  If  the  quantity  of  interest  involves  modeling 
for  each  monitoring  event  (e.g.,  contaminant  mass  based  on  interpolation)  that  modeling 
is  provided  by  Model  Builder. 
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Figure  2-1 .  General  Flowehart  of  Software  Applieation 


Although  the  software  provides  speeific  results  as  output,  there  is  still  an  aspeet  of  interpretation 
required  by  an  analyst,  as  illustrated  on  Figure  2-1 .  With  respeet  to  the  data  redundancy 
evaluation,  the  software  produces  an  optimal  tradeoff  curve  (example  provided  on  Figure  2-2), 
and  the  analyst  must  choose  specific  sampling  plans  along  the  tradeoff  curve  for  further 
evaluation.  For  each  number  of  wells,  SO  evaluates  many  potential  sampling  plans;  the 
sampling  plans  with  the  least  errors  are  those  shown  in  the  tradeoff  curves. 


Figure  2-2.  Example  of  a  Tradeoff  Curve  for  Spatial  Optimization 
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The  Summit  Software  calculates  the  “error”  as  a  dimensionless  parameter,  using  a  tool  called  the 
“Cutoff  Error  Calculator”,  which  is  described  in  detail  in  the  Technical  Report.  The  error  is  an 
indicator  of  the  maximum  difference  between  an  interpolated  value  and  an  actual  value  at  any 
location  where  a  sample  is  eliminated. 

With  respect  to  the  detection  of  anomalies  with  DT,  the  software  identifies  values  in  recently 
collected  data  that  are  “out-of-bounds”  and  provides  a  graph  of  concentration  versus  time  for 
visual  review.  The  analyst  must  then  determine  if  any  response  or  action  is  appropriate,  such  as 
correction  of  an  erroneous  laboratory  report  or  further  investigation  of  a  potential  new  source  of 
contamination.  Thus,  the  corrective  action  or  response  referred  to  in  the  figure  can  take  many 
forms  depending  on  the  nature  of  the  anomaly. 

The  Summit  tools  were  built  upon  technology  development  and  research  at  the  University  of 
Illinois  from  1997  -  2004.  Reed  et  al.  (2003),  with  funding  from  an  Environmental  Protection 
Agency  STAR  Eellowship,  created  an  automated  methodology  for  setting  parameter  values  for 
the  Nondominated  Sorted  Genetic  Algorithm-II  (NSGA-II).  A  Technology,  Research,  Education, 
and  Commercialization  Center  (TRECC,  http://www.trecc.org/)  project  funded  by  the  Office  of 
Naval  Research  implemented  NSGA-II  with  the  automated  parameter-setting  methodology  in  the 
Data  to  Knowledge  (D2K)  software  development  and  data  mining  framework  created  by  the 
National  Center  for  Supercomputing  Applications  (NCSA).  The  resulting  software,  called 
Evolutionary  Multi-objective  Optimizer  (EMO),  was  further  developed  with  funding  from 
BP/Atlantic  Richfield  to  Barbara  Minsker,  Riverglass  Inc.,  and  Hazard  Management  Systems 
Inc.  (HMSI).  HMSI  created  Sampling  Optimizer  as  a  specific  application  of  EMO  to  long-term 
monitoring  optimization,  along  with  Data  Tracker. 

Two  case  studies  using  Sampling  Optimizer  were  completed  in  July  2004.  Site  A  had  36 
sampling  locations  and  the  optimization  focused  on  BTEX  characterization,  while  Site  B  had  80 
sampling  locations  and  focused  on  benzene  characterization.  Each  study  identified  roughly  23% 
redundancy  in  the  sampling  locations  eligible  for  removal. 

HMSI  was  purchased  by  Summit  Envirosolutions  in  December  2006,  who  have  continued  to 
improve  and  develop  and  refine  the  software  and  documentation  in  concert  with  BP  and  this 
ESTCP  project.  During  this  ESTCP  project  several  features  were  added  or  improved  based  on 
feedback  from  the  project  team.  Summit  is  currently  performing  several  additional  case  studies 
to  further  evaluate  and  demonstrate  the  effectiveness  of  the  software;  summaries  of  these  results 
will  be  posted  on  the  Sampling  Optimizer  website. 

The  Summit  Software  is  expected  to  provide  the  following  benefits  related  to  its  expected 
applicability: 

•  Significant  cost  savings  are  expected  by  eliminating  redundant  data  collection.  Costs 
could  be  reduced  for  sampling  labor,  laboratory  analysis,  data  management,  etc. 
Experience  suggests  that  eliminating  redundant  sampling  points  can  save  $500-$  1,000  or 
more  per  sample  in  labor  and  analysis.  Additionally,  the  Sampling  Optimizer  will  help 
managers  identify  tradeoffs  among  multiple  monitoring  objectives,  including  identifying 
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when  further  monitoring  expenditures  will  likely  result  in  minimal  benefits. 

•  New  data  ean  be  given  an  initial  assessment  for  signifieant  deviations  and  other  features 
of  interest  with  a  eonsiderable  reduetion  in  labor.  Currently,  at  most  sites  even  a  quiek 
visual  sean  for  a  few  eonstituents  at  a  few  key  wells  ean  require  several  hours  for  an 
analyst.  Thorough  statistieal  tests  eould  take  weeks  of  labor,  but  ean  be  readily 
performed  by  Data  Traeker.  This  may  enable  earlier  deteetion  and  eorreetion  of  potential 
problems  and/or  faster  identifieation  of  signifieant  ehanges  in  the  physieal  system,  whieh 
results  in  higher  eertainty  of  attaining  proteetiveness.  This  benefit  will  beeome  even 
greater  as  emerging  sensor  teehnologies  produee  larger  volumes  of  data  to  be  analyzed 
and/or  more  faeilities  move  into  LTM  in  post-elosure  and/or  passive  remediation 
seenarios. 


2.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

Related  software  and  methods  for  LTM  design  exist,  as  follows: 

•  Geostatistieal  paekages  are  widely  available  (e.g.,  Surfer,Geo-EAS)  but  these  paekages 
ereate  only  interpolation  models  and  do  not  perform  data  traeking  or  optimization. 

•  Based  on  eurrent  information,  GTS  (Cameron  and  Hunter,  2002),  the  3-tiered  monitoring 
optimization  approaeh  by  Parsons  (Nobel,  2003),  MAROS  (Aziz,  et  ah,  2003),  and  Cost 
Effeetive  Sampling  (Johnson,  et  ah,  1996)  perform  various  spatial  and  temporal 
redundaney  analyses  for  ETM.  However,  they  do  not  perform  data  traeking  for  site-wide 
targets  and  do  not  use  mathematieal  optimization.  Instead,  they  use  heuristie  (“rule-of- 
thumb”)  approaehes  for  identifying  whieh  samples  to  remove,  whieh  may  not  identify  the 
optimal  sampling  plan  to  best  meet  the  site-speeifie  objeetives.  Moreover,  most  of  these 
methods  are  not  yet  available  as  supported  software  paekages  (with  the  exeeption  of 
MAROS  and  a  limited  version  of  GTS).  These  methods  do  not  optimize  based  on 
removing  groupings  of  samples,  and  do  not  yield  tradeoff  eurves  based  on  the  results  of 
multi-objeetive  mathematieal  optimization. 

•  Herrera  and  Finder  (1998)  and  Rizzo  et  al.  (2000)  have  used  Kalman  filters  for  ETMO 
and  model  updating.  These  approaehes  require  the  use  of  transport  models  for  the 
analysis,  whieh  most  DOD  sites  do  not  have.  When  transport  models  are  available, 
Kalman  filters  ean  use  available  data  to  update  the  models  as  new  data  beeome  available. 
Herrera  and  Finder’s  approaeh  uses  this  eapability  to  identify  the  next  loeation  that 
should  be  sampled,  seleeting  the  loeation  with  the  most  uneertainty  in  the  model 
predietions,  after  eaeh  sample  is  eolleeted  with  an  event.  This  approaeh  assumes  that  data 
at  a  site  are  eolleeted  sequentially,  with  enough  time  between  eaeh  sample  to  analyze  the 
previous  result  and  use  it  to  determine  the  next  sample  loeation,  whieh  is  not  always  the 
ease.  Additionally,  the  sequential  sampling  approaeh  is  not  a  global  algorithm  that 
identifies  the  best  set  of  loeations  to  maximize  the  overall  reduetion  in  uneertainty.  Rizzo 
et  al.  eouple  the  Kalman  filter  with  simulated  annealing,  a  global  optimization  approaeh, 
but  to  our  knowledge  is  not  available  as  a  software  paekage. 
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Some  of  the  advantages  of  the  Summit  Software  demonstrated  in  this  ESTCP  projeet  are  listed 
below: 


•  The  Summit  Software  is  the  only  user-friendly  software  available  (i.e.,  not  a  researeh 
code)  that  performs  monitoring  optimization  with  mathematical  algorithms  that  provide 
optimal  or  near-optimal  solutions  with  high  probability. 

•  A  major  advantage  of  the  optimization  approach  utilized  in  the  Summit  Software  is  that  it 
allows  sampling  redundancy  to  be  evaluated  on  a  system-wide  basis,  identifying  optimal 
solutions  with  one,  two,  three,  etc.  locations  removed,  rather  than  on  a  well-by  well  basis 
such  as  the  redundancy  analysis  employed  in  MAROS. 

•  It  is  the  only  LTMO  software  available  that  enables  users  to  select  multiple  site-specific 
monitoring  objectives  for  the  redundancy  analysis,  thus  allowing  the  tradeoff  between  the 
number  of  samples  and  the  resulting  error  to  be  rigorously  evaluated. 

•  Visualizations  of  the  plume  for  the  baseline  plan  with  all  samples  versus  improved  plans 
with  reduced  number  of  samples  are  created  within  the  software,  which  is  not  the  case 
with  MAROS. 

•  It  is  the  only  LTMO  software  currently  available  to  incorporate  data  tracking  capabilities 
to  semi-automatically  identify  unexpected  values  in  recently  collected  data. 

On  the  other  hand,  the  Summit  Software  does  not  incorporate  some  features  that  are  available  in 
other  software.  For  example,  (1)  the  Summit  Software  has  fewer  types  of  interpolation  models 
than  most  available  geostatistical  packages  (although  the  interpolation  models  available  in  the 
Summit  Software  are  standard  and  well  accepted);  (2)  the  data  tracking  routine  in  the  Summit 
Software  does  not  include  trend  analysis  (such  as  performed  by  MAROS)  to  indicate  if 
concentration  trends  at  individual  wells  are  increasing,  decreasing,  or  stable;  and  (3)  the  Summit 
Software  does  not  incorporate  transport  models  as  does  the  Kalman  filter  approach. 

Although  the  Summit  Software  performs  spatiotemporal  optimization,  it  does  so  using  sequential 
spatial  interpolation  and  does  not  include  intra-well  temporal  interpolation,  which  likely  makes 
the  results  of  the  spatiotemporal  optimization  overly  conservative  (this  is  discussed  in  more 
detail  later  in  the  report).  Another  limitation  is  that  the  mass  and  mass  flux  tracking  features  in 
the  software  are  impacted  by  differences  in  the  number  of  sampling  locations  in  each  event.  This 
is  related  to  the  fact  that  the  software  does  not  perform  temporal  interpolation  or  extrapolation  to 
fill  in  missing  values  in  events  where  specific  wells  are  not  sampled.  It  is  noted  that  MAROS 
has  the  same  limitation  in  this  regard. 

The  Summit  Software  is  designed  for  sites  meeting  the  following  criteria  which  represent 
limitations  to  application  of  the  software;  other  approaches  or  software  products  designed  for 
similar  purposes  will  undoubtedly  have  similar  limitations: 
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•  First,  for  Model  Builder  and  Sampling  Optimizer  to  provide  reliable  results  the  site 
should  be  in  a  LTM  situation,  whieh  may  include  ongoing  active  or  passive  remediation. 
This  implies  that  the  groundwater  chemistry  should  be  expected  to  change  smoothly  if  at 
all  in  the  foreseeable  future,  whether  or  not  remediation  activities  are  underway,  and  that 
no  new  sources  or  potential  releases  are  anticipated. 

•  The  site  should  have  an  adequate  data  history  available  that  is  representative  of  its  current 
groundwater  chemistry  and  status.  If  spatiotemporal  redundancy  evaluation  using  Model 
Builder  and  Sampling  Optimizer  is  to  be  performed,  a  site  should  generally  have  at  least 
four  observations  obtained  over  a  period  of  at  least  two  years  at  most  wells  under 
consideration  (spatial  redundancy  evaluation  has  no  such  requirement),  and  eight 
observations  is  preferred.  Strictly  speaking,  when  using  Sampling  Optimizer  for  spatial 
optimization,  only  one  data  value  per  well  for  each  primary  COC  is  needed  for  using  the 
software,  but  a  more  extended  data  history  is  preferable  to  verify  the  presence  of  the 
requisite  LTM  situation.  For  DT  the  software  requires  an  absolute  minimum  of  four 
background  observations  per  COC  per  well;  more  observations  are  desirable. 

•  The  specific  number  of  wells  needed  depends  on  the  site  complexity.  In  general  there 
should  be  at  least  twenty  monitoring  wells  in  order  to  anticipate  significant  cost 
reductions  from  a  spatial  redundancy  analysis  (otherwise  a  significant  percentage 
reduction  in  sampling  locations  would  only  yield  a  modest  cost  savings).  If  one  is 
interested  only  in  using  Data  Tracker,  an  adequate  data  history  for  each  constituent  of 
interest  at  each  well  is  needed  (at  least  four  previous  samples  to  use  as  background  data), 
but  there  is  no  requirement  for  a  minimum  number  of  wells. 

•  There  will  generally  be  one  or  at  most  a  few  primary  COCs  with  respect  to  which  the 
modeling  and  optimization  for  redundancy  are  performed.  Data  Tracker  is  not  limited  to 
the  COCs  used  by  Model  Builder  and  Sampling  Optimizer. 

•  For  efficient  use  of  Data  Tracker,  future  routine  monitoring  data  should  be  made 
available  in  compatible  electronic  form.  This  may  be  expedited  by  constructing  a  “data 
bridge”  mechanism  is  (i.e.,  a  program  that  reads  data  in  one  format  and  rewrites  the  data 
into  a  desired  format). 

•  One  of  the  assumptions  for  applying  the  Summit  Software  is  there  are  no  major 
discontinuities  in  the  specific  aquifer  being  evaluated  with  respect  to  hydraulic 
connection.  Highly  fractured  media  would  not  meet  that  assumption.  This  limitation 
would  of  course  be  true  of  LTMO  software  in  general,  not  just  the  Summit  Software. 
Similarly,  sites  with  extremely  large  contrasts  in  hydraulic  conductivity  (e.g.,  preferential 
pathways)  might  impact  the  application  of  the  software. 

The  limitations  listed  above  pertain  to  the  overall  technology.  Several  additional  limitations 
within  the  software  functionality  observed  during  testing  are  discussed  later  in  this  report. 
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3.0  PERFORMANCE  OBJECTIVES 


Table  3-1  summarizes  the  performanee  objeetives  for  evaluating  the  Summit  Software  provided 
in  the  Teehnology  Demonstration  Plan.  To  avoid  repetition,  a  detailed  diseussion  is  provided  for 
eaeh  performanee  objeetive  in  Seetion  6.0. 


Table  3-1.  Performanee  Objeetives 


Type  of 
Performance 
Objective 

Performance 

Criteria 

Expected  Performance  (Metric) 

Performance 
Objective  Met? 

Qualitative 

User  functionality 
(primary) 

The  Summit  Software  has  an  acceptable 
learning  curve  (e.g.,  1-2  days)  that  will  not 
discourage  prospective  users  and  allow 
users  to  achieve  the  intended  objectives. 

YES 

Software  reliability 
(primary) 

The  Summit  Software  has  no  significant 
errors  or  bugs  remaining  by  the  end  of  this 
project. 

YES 

Model  Builder 

performance 

(primary) 

Model  Builder  provides  a  model  of  spatial 
and/or  temporal  variation  for  each  primary 
constituent  of  concern  at  each  site  that  is 
adequate  given  the  available  data. 

YES 

(spatial) 

PARTIALLY 

(temporal) 

Sampling  Optimizer 

performance 

(primary) 

Sampling  Optimizer  provides  reasonable 
trade-off  curves  allowing  site  personnel  and 
other  professionals  to  easily  identify 
optimal  monitoring  program  choices. 

YES 

(spatial) 

PARTIALLY 

(spatiotemporal) 

Data  Tracker 

performance 

(primary) 

Data  Tracker  enables  the  easy  incorporation 
of  site-specific  monitoring  and  remediation 
expectations  and  data  objectives  along  with 
historical  data. 

PARTIALLY 

Regulatory 

acceptance 

(primary) 

Results  of  this  ESTCP  dem/val  will  be 
persuasive  to  regulatory  personnel. 

YES 

Comparison  with 

MAROS 

(secondary) 

The  Summit  Software  will  be  found  to  be  at 
most  modestly  more  difficult  to  learn  to  use, 
consistent  with  being  much  more  flexible  in 
incorporating  site-specific  monitoring 
objectives. 

There  are  no  pre-conceptions  regarding  the 
comparisons  of  optimization 
recommendations  to  be  expected  from  the 
two  software  products. 

YES 
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Type  of 
Performance 
Objective 

Performance 

Criteria 

Expected  Performance  (Metric) 

Performance 
Objective  Met? 

Quantitative 

Model  Builder 

performance 

(primary) 

Model  Builder  provides  a  model  of  spatial 
and/or  temporal  variation  for  each  primary 
constituent  of  concern  at  each  site  that  is 
adequate  given  the  available  data. 

YES 

(spatial) 

PARTIALLY 

(temporal) 

Sampling  Optimizer 

performance 

(primary) 

Optimized  programs  identified  by  Sampling 
Optimizer  in  fact  permit  cost  reductions 
with  acceptable  losses  of  information,  if 
appropriate,  as  anticipated  for  the  large 
majority  of  DoD  sites. 

YES 

Data  Tracker 

performance 

(primary) 

Data  Tracker  responds  appropriately  to 
artificially  induced  anomalies  of  interest  for 
the  particular  site. 

YES 

Comparison  with 

MAROS 

(secondary) 

The  Summit  Software  will  be  found  to  be  at 
most  modestly  more  difficult  to  learn  to  use, 
consistent  with  being  much  more  flexible  in 
incorporating  site-specific  monitoring 
objectives. 

If  both  products  are  able  to  accept  the  same 
goals  and  constraints,  results  will  be  similar 
but  slightly  different  due  to  small 
differences  due  to  different  optimization 
methodologies. 

YES 

There  are  no  pre-conceptions  regarding  the 
comparisons  of  optimization 
recommendations  to  be  expected  from  the 
two  software  products. 
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4.0  SITE  DESCRIPTION 


Three  DoD  demonstration  sites  were  seleeted.  Table  4-1  provides  a  summary  of  the 
demonstration  sites. 


T able  4- 1 .  Summary  of  Demonstration  Sites 


Norfolk  Naval  Station 
Camp  Allen  Landfill 

Former  George  AFB 
(GAFB)  OUl 

Former  Nebraska 
Ordnance  Plant 
(NOP)  OU2 

Agency 

Navy 

Air  Eorce 

Army 

Location 

Norfolk,  VA 

Victorville,  CA 

Mead,  NE 

Geographic 

Location 

East 

(coastal) 

West 

(arid) 

Midwest 

(plains) 

Remediation 

System 

P&T  with  air  stripping 
for  hydraulic 
containment 

P&T  started  in  1991 
and  shut  down  since 
2003 

P&T  with  10  extraction 
wells 

cl2DCE,  TCE,  VC 

TCE 

TCE  and  RDX 

Aquifers 

Evaluated 

Shallow  and  deep 
aquifers 

Upper  aquifer 

Shallow,  intermediate, 
and  deep  aquifers 

Sampling 

Frequency 

Annual 

Semi-annual 

Varies  by  well 

Monitoring 

Network 

~70 

~50 

-220 

4, 1  SITE  LOCATION  AND  HISTORY 

Loeations  are  summarized  in  Table  4-1.  Loeation  maps  are  provided  in  the  Teehnieal  Report. 


4.2  SITE  GEOLOGY/HYDROGEOLOGY 

The  number  of  aquifers  evaluated  at  eaeh  site  is  summarized  in  Table  4-1.  Details  regarding 
stratigraphy  and  groundwater  flow  patterns  at  eaeh  site  are  provided  in  the  Teehnieal  Report. 


4,3  CONTAMINANT  DISTRIBUTION 

The  primary  eontaminants  of  eoncern  (COCs)  at  each  site,  and  the  approximate  number  of  wells 
comprising  the  monitoring  network  at  each  site,  are  summarized  in  Table  4-1.  Plume  maps  are 
provided  in  the  Technical  Report. 
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5.0  TESTING  DESIGN 


5.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

Initially,  information  was  obtained  about  a  number  of  eandidate  sites.  Three  sites  were  seleeted 
based  on  eonsiderations  ineluding  monitoring  and  regulatory  status,  data  availability,  and 
representation  aeross  the  DoD.  The  following  general  approaeh  was  applied  for  eaeh  of  the 
demonstration  sites  seleeted: 

•  The  ESTCP  projeet  team  obtained  preliminary  information  from  the  Site  team  for  review 
prior  to  site  visit  (e.g.,  reports  deseribing  site  eoneeptual  model  and  site  history) 

•  The  ESTCP  projeet  team  eondueted  a  site  visit  to  present  an  overview  of  the  projeet  and 
to  reeeive  input  regarding  the  optimization  formulation  from  the  Site  team 

•  The  ESTCP  projeet  team  developed  a  preliminary  optimization  formulation,  provided  it 
to  the  Site  team  for  review,  and  then  finalized  the  optimization  formulation  based  on 
feedbaek  from  the  Site  team 

•  The  Site  team  then  provided  the  most  updated  version  of  historieal  sampling  data  to 
EnviroStat  in  eleetronie  format 

•  EnviroStat  then  performed  the  following  aetivities: 

o  Sereened  historieal  data  to  determine  if  any  obvious  data  quality  issues  were 
evident 

o  Attempted  to  resolve  any  data  quality  issues  with  the  Site  team 

o  Reserved  the  last  year  of  sampling  data  (whieh  would  be  used  later  for  validation 
of  Sampling  Optimizer  results  as  well  as  for  evaluation  of  Data  Traeker) 

o  Provided  GeoTrans  with  Comma  Separated  Variable  (CSV)  files,  not  ineluding 
the  reserved  data,  that  eould  used  as  input  to  the  software  for  evaluation  of  the 
Sampling  Optimizer  funetionality 

o  Created  five  alternate  versions  of  the  reserved  data  that  ineorporated  artifieial 
anomalies  based  on  diseussions  with  the  Site  team  about  seenarios  of  potential 
interest 

•  GeoTrans  applied  the  Summit  Software  to  evaluate  data  redundaney  (i.e.,  the  Sampling 
Optimizer  module,  in  eonjunetion  with  Model  Builder),  and  provided  the  ESTCP  projeet 
team  with  a  preliminary  write-up  of  results 


12 


•  EnviroStat  then  provided  GeoTrans  with  the  six  versions  of  the  reserved  (i.e.,  eurrent) 
data  for  input  to  Data  Traeker  (i.e.,  one  was  the  aetual  version,  and  the  other  five  had 
artifieial  anomalies  as  well  as  “tweaked”  values  so  that  identifieation  of  the  aetual 
version  by  GeoTrans  would  be  diffieult) 

•  GeoTrans  applied  the  Summit  Software  to  evaluate  the  Data  Traeker  funetionality,  and 
provided  a  summary  of  Data  Traeker  results  to  EnviroStat 

•  EnviroStat  then  revealed  whieh  of  the  six  versions  of  the  reserved  data  was  the  aetual 
version,  and  GeoTrans  used  that  dataset  to  perform  validation  of  the  Sampling  Optimizer 
results  (i.e.,  used  the  more  reeent  data  to  make  plume  maps  using  the  baseline  well 
loeations,  and  plume  maps  based  on  the  optimized  sampling  plans,  to  see  if  the  maps 
based  on  the  optimized  sampling  plans  appeared  reasonable) 

•  GeoTrans  provided  the  ESTCP  team  with  a  write-up  of  results  and  eonelusions, 
ineorporating  seetions  provided  by  EnviroStat  regarding  data  preparation  and 
interpretation  of  Data  Traeker  results 

•  After  review  by  the  ESTCP  team,  the  write-up  was  finalized  and  forwarded  to  the  Site 
team  for  their  review  and  feedbaek,  followed  by  a  presentation  of  results  to  the  Site  team 
via  eonferenee  eall  (Camp  Allen  site,  NOP  site)  or  in-person  meeting  (GAFB  site). 

In  addition  to  the  analysis  for  eaeh  site  deseribed  above,  the  following  aetivities  were  also  part  of 
the  projeet  design: 

•  Eor  one  of  the  three  sites  (GAEB),  additional  validation  of  results  was  performed  based 
on  one  year  of  sampling  eondueted  subsequent  to  the  original  set  of  reserved  data 

•  Eor  one  of  the  three  sites  (Camp  Allen)  the  MAROS  software  was  also  applied,  so  that 
funetionality  and  results  (to  the  extent  possible)  eould  be  eompared  (note  that  GTS  and 
MAROS  will  also  be  applied  at  the  NOP  site  in  a  separate  effort  outside  of  this  projeet) 

•  EPA  Region  V  arranged  for  their  eontraetor  to  apply  the  software  at  one  of  their  sites, 
and  provided  feedbaek  to  the  ESTCP  projeet  team  by  filling  out  a  questionnaire  and 
preparing  a  brief  summary  report 

Throughout  the  projeet  the  Summit  Software  was  applied  by  a  mid-level  GeoTrans  engineer  with 
no  previous  experienee  with  ETMO  software,  rather  than  by  the  software  developer.  In  this 
manner,  the  software  demonstration  provided  a  realistie  evaluation  of  its  usability  by  a  typieal 
DoD  eontraetor  with  no  prior  experienee  with  the  software.  Before  the  first  site  was  evaluated, 
the  GeoTrans  analyst  was  provided  with  basie  training  for  both  the  Summit  Software  and  the 
MAROS  eode.  The  subsequent  two  sites  were  evaluated  by  a  different  mid-level  GeoTrans 
analyst  who  was  not  provided  any  training  on  either  software. 
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5.2  BASELINE  CHARACTERIZATION 


The  steps  in  the  experimental  design  that  might  be  eonsidered  “baseline  eharacterization”  are 
those  assoeiated  with  the  developing  the  optimization  formulation  for  eaeh  site,  and  obtaining 
and  preparing  the  data  for  eaeh  site.  Those  items  are  deseribed  in  detail  in  the  Teehnieal  Report. 


5.3  TREATABILITY  OR  LABORATORY  STUDY  RESULTS 

These  items  do  not  apply  to  this  ESTCP  project. 


5.4  FIELD  TESTING 

The  schedule  for  testing  of  the  software  is  summarized  in  Table  5-1. 


Table  5-1.  Schedule  for  Testing  the  Software 


2007 

2008 

J 

F 

M 

A 

M 

J 

J 

A 

S 

o 

N 

D 

J 

F 

M 

A 

M 

J 

J 

A 

S 

o 

N 

D 

Camp  Allen  Site: 

-  Site  Visit 

-  Formulation 

-  Software  Testing 

-  Apply  MAROS 

GAFB  Site: 

-  Site  Visit 

-  Formulation 

-  Software  Testing 

-  Follow-up  Evaluation 

■ 

■ 

NOP  Site: 

-  Site  Visit 

-  Formulation 

-  Software  Testing 

EPA  Testing  at  One  Site 

{this  gap  is  intentional} 
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The  analysis  for  each  demonstration  site  incorporated  variations  that  increased  the  robustness  of 
the  testing.  These  variations  included  the  following; 

•  Camp  Allen  Site 

o  Evaluated  two  aquifers  (shallow  and  deep) 

o  Evaluated  all  six  combinations  of  interpolation  method  and  transformation  type  in 
Model  Builder 

o  Evaluated  all  six  combinations  of  interpolation  method  and  transformation  type 
for  spatial  redundancy  evaluation,  and  two  of  the  combinations  (kriging-quantile 
and  IDW-quantile)  for  spatiotemporal  redundancy  evaluation 

o  Evaluated  three  COCs  simultaneously  for  redundancy  evaluation 

o  Evaluated  different  values  for  Population  Size  in  the  GA  in  SO  for  both  spatial 
and  spatiotemporal  redundancy  evaluation 

•  GAEB  Site 

o  Evaluated  one  aquifer  (upper) 

o  Evaluated  all  six  combinations  of  interpolation  method  and  transformation  type  in 
Model  Builder 

o  Evaluated  two  combinations  of  interpolation  method  and  transformation  type  for 
spatial  redundancy  evaluation  (kriging-quantile  and  IDW-quantile),  and  one 
combination  for  spatiotemporal  redundancy  evaluation  (kriging-quantile) 

o  Evaluated  two  variations  of  the  baseline  data:  Dataset  A  had  55  wells  and  Dataset 
B  had  47  wells  (eight  wells  that  had  atypical  water  levels  and  screened  intervals 
were  removed  from  Dataset  B) 

o  Evaluated  different  combinations  of  values  for  Population  Size  and  Number  of 
Generations  in  the  GA  in  both  spatial  and  spatiotemporal  redundancy  evaluation 

o  Evaluated  the  Mass  Metric  and  Mass  Elux  functionality  for  two  cases:  one  with 
uneven  distribution  of  samples  per  event,  and  one  where  missing  data  were  filled 
in  manually  (outside  the  software)  based  on  temporal  interpolation 

o  Eor  spatiotemporal  redundancy  evaluation,  evaluated  use  of  the  original  dataset 
versus  a  modified  dataset  where  missing  data  were  filled  in  manually  (outside  the 
software)  based  on  temporal  interpolation 
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o  For  both  spatial  and  spatiotemporal  redundancy  evaluation,  utilized  three  different 
values  for  cut-off  concentration  between  low  concentration  (i.e.,  plume  boundary) 
and  high  concentration  (i.e.,  plume  interior)  areas 

•  NOP  site 

o  Evaluated  three  aquifers  (shallow,  intermediate,  and  deep) 

o  Evaluated  all  six  combinations  of  interpolation  method  and  transformation  type  in 
Model  Builder 

o  Evaluated  all  six  combination  of  interpolation  method  and  transformation  type 
for  spatial  redundancy  evaluation 

o  Evaluated  the  difference  between  considering  multiple  COCs  (i.e.,  TCE  and 
RDX)  simultaneously  versus  independently  for  the  redundancy  evaluation 

In  addition,  for  the  DT  evaluation  at  each  of  the  three  sites,  artificial  anomalies  were  added  to  the 
actual  reserved  data  by  EnviroStat.  These  anomalies  followed  a  number  of  scenarios  that  had 
been  discussed  with  the  Site  Teams,  such  as 

•  Abnormally  high  concentrations  in  a  particular  area  of  the  site  due  to  new  contaminant 
source  and/or  plume  migration 

•  Abnormally  high  concentrations  at  individual  wells  for  no  apparent  reason 

•  Abnormally  low  values  for  a  number  of  samples  in  the  same  event  due  to  a  systematic  lab 
error 

•  Switched  samples  (e.g.,  bottles  labeled  incorrectly  and/or  reported  incorrectly  by  the  lab) 

•  Eaboratory  cross-contamination 

•  Database  errors 

In  addition,  it  turned  out  that  some  of  these  anomalies  were  present  in  the  actual  data  from  two  of 
the  sites. 

During  the  application  of  the  software  at  the  first  two  demonstration  sites,  several  minor  bugs 
were  detected  and  reported  to  the  software  developer.  Eor  example,  in  DT  there  was  an  issue 
with  the  plotting  of  the  y-axis  data  labels  that  was  identified.  By  the  end  of  the  demonstration 
product,  no  remaining  bugs  were  known  to  exist.  During  this  demonstration  project  several 
software  features  were  added  or  improved;  those  are  detailed  in  the  Technical  Report.  These 
were  not  due  to  bugs;  rather,  they  represent  evolution  of  software  features. 
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The  detailed  results  obtained  from  the  field  testing  are  presented  in  the  Technical  Report.  Some 
examples  of  the  results  are  presented  below.  Figure  5-1  is  an  example  tradeoff  curve  from  the 
GAFB  spatial  redundancy  analysis  (the  details  regarding  how  error  is  calculated,  and  how  the 
optimization  problems  were  formulated,  are  provided  in  the  Technical  Report).  Each  diamond 
represents  the  optimal  monitoring  design  for  a  given  level  of  expenditure  (i.e.,  number  of  wells). 
Such  solutions  are  optimal  (non-dominated)  because  no  other  possible  solution  is  superior  in 
both  objectives  simultaneously.  The  advantage  of  using  GAs  to  solve  multi-objective  problems 
is  that  the  entire  set  of  such  optimal  solutions  can  be  generated  in  a  single  optimization  run.  In 
this  case  the  user  selected  two  plans  along  the  tradeoff  curve  (plans  97  and  14)  for  further 
evaluation  (comparing  plume  maps  generated  by  the  software  for  those  plans  to  each  other,  and 
to  the  baseline  plan  with  all  sampling  locations).  The  user  can  select  any  plan  along  the  tradeoff 
curve  for  further  evaluation.  Plans  further  to  the  left  will  have  more  wells  remaining  (i.e.,  fewer 
wells  eliminated)  and  less  maximum  value  of  error  at  any  of  the  removed  locations. 


Figure  5-1.  Example  Tradeoff  Curve  for  Spatial  Redundancy  Evaluation,  GAEB  Site 
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Eigure  5-2  is  an  example  of  plume  maps  generated  by  the  software  for  two  plans  selected  from 
the  tradeoff  curve  above.  In  this  case  the  plume  maps  illustrate  that  the  selected  plans  with 
reduced  number  of  sampling  locations  provide  for  a  generally  similar  graphical  representation  of 
the  plume. 
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Figure  5-2.  Example  of  Plume  Maps  for  Plans  Selected  from  Tradeoff  Curve 


All  sampling  locations 
(55  wells) 


Plan  97 
(41  wells) 
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Note: 

1.  The  symbol  indicates  wells  that  are  recommended  to  be  removed  by  the  Optimizer,  while  the  symbol  “O 
denotes  wells  that  are  recommended  to  keep. 
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It  is  difficult  to  fully  summarize  the  results  for  the  three  demonstration  sites,  since  the  software 
produces  a  tradeoff  curve  for  each  optimization  simulation  which  provides  a  family  of  optimal 
solutions.  In  Tables  5-2  to  5-4,  the  savings  versus  the  baseline  data  set  are  presented  for 
different  values  of  normalized  error  along  the  tradeoff  curve. 


Table  5-2.  Summary  of  Redundancy  Evaluation  Results,  Camp  Allen  Site 


Optimization 

#  of  samples  in 
the  baseline 
model 

#  of  samples  with 
the  max  error  per 
COC  of  0.5 

#  of  samples  with 
the  max  error  per 
COC  of  1.0 

#  of  samples  with 
the  max  error  per 
COC  of  1.5 

Shallow  Aquifer 

Spatial 

42 

17  (59.5%  saving) 

16  (61.9%  saving) 

14  (66.6%  saving) 

Spatiotemporal 

N/A 

N/A 

N/A 

N/A 

Deep  Aquifer 

Spatial 

31 

28  (9.7%  saving) 

21  (32.2%  saving) 

21  (32.2%  saving) 

Spatiotemporal 

21 

20.2  (3.8%  saving) 

17.7  (15.7%  saving) 

17.5  (16.7%  saving) 

*note:  number  of  samples  is  the  number  of  wells  (spatial)  or  the  number  of  samples  per  year  (spatiotemporal) 
Aquifers:  Shallow  and  Deep  (evaluated  separately) 

COCs  Evaluated:  cl2DCE,  TCE,  VC  (evaluated  simultaneously,  errors  for  each  COC  added  together) 
Model:  Kriging  interpolation  with  quantile  data  transformation 
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Table  5-3.  Summary  of  Redundancy  Evaluation  Results,  GAFB  Site 


Cutoff 

Value 

Optimizatiou 

#of 

samples  iu 
the  baseliue 
model 

#  of  samples  with 
the  max  error  0.5 

#  of  samples  with 
the  max  error  1.0 

#  of  samples  with  the 
max  error  1.5 

Upper  Aquifer  with  Dataset  A 

25 

Spatial 

55 

40  (27.3%  saving) 

34  (38.2%  saving) 

30  (45.5%  saving) 

50 

55 

42  (23.6%  saving) 

37  (32.7%  saving) 

31  (43.6%  saving) 

100 

55 

41  (25.5%  saving) 

36  (34.5%  saving) 

32  (41.8%  saving) 

25 

Spatio- 

temporal 

108 

102  (5.6%  saving) 

96.33  (10.8%  saving) 

93.08  (13.8%  saving) 

50 

108 

104  (3.7%  saving) 

96.33  (10.8%  saving) 

92.83  (14.0%  saving) 

100 

108 

102  (5.6%  saving) 

98.08  (9.2%  saving) 

94.83  (12.2%  saving) 

Upper  Aquifer  with  Dataset  B** 

25 

Spatial 

47 

36  (23.4%  saving) 

30  (36.2%  saving) 

28  (40.4%  saving) 

50 

47 

36  (23.4%  saving) 

31  (34.0%  saving) 

28  (40.4%  saving) 

100 

47 

36  (23.4%  saving) 

32  (31.9%  saving) 

29  (38.3%  saving) 

25 

Spatio- 

temporal 

92 

88  (4.3%  saving) 

82.53  (10.3%  saving) 

79.08  (14.0%  saving) 

50 

92 

88  (4.3%  saving) 

82.53  (10.3%  saving) 

79.08  (14.0%  saving) 

100 

92 

88  (4.3%  saving) 

82.53  (10.3%  saving) 

79.33  (13.8%  saving) 

*note:  number  of  samples  is  the  number  of  wells  (spatial)  or  the  number  of  samples  per  year  (spatiotemporal) 
**  Eight  wells  (MW-102,  MW-104,  NZ-06,  NZ-10,  NZ-20,  NZ-30,  NZ-3 1,  and  NZ-32)  identified  not  to  be 


representative  of  aquifer  characteristics  are  excluded  from  Dataset  B. 

Cutoff  value  is  used  by  the  software  to  differentiate  areas  of  lower  concentration  from  areas  of  higher 
Concentration  when  calculating  error  (errors  in  low  concentration  areas  are  given  more  weight) 

Aquifers:  Shallow 

COCs  Evaluated:  TCE 

Model:  Kriging  interpolation  with  quantile  data  transformation 
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Table  5-4.  Summary  of  Redundancy  Evaluation  Results,  NOP  Site 


Optimization 

#  of  samples  in 
tbe  baseline 
model 

#  of  samples  with 
tbe  max  combined 
error  of  0.5 

#  of  samples  with 
tbe  max  combined 
error  of  1.0 

#  of  samples  with 
tbe  max  combined 
error  of  1.5 

Shallow  Aquifer  (25  wells  are  non-removable) 

Spatial 

81 

54  (33.3%  saving) 

50  (38.3%  saving) 

47  (42.0%  saving) 

Spatiotemporal 

N/A 

N/A 

N/A 

N/A 

Intermediate  Aquifer  (25  wells  are  non-removable) 

Spatial 

84 

48  (42.9%  saving) 

43  (48.8%  saving) 

42  (50.0%  saving) 

Spatiotemporal 

N/A 

N/A 

N/A 

N/A 

Deep  Aquifer  (22  wells  are  non-removable) 

Spatial 

56 

35  (37.5%  saving) 

33  (41.1%  saving) 

32  (42.9%  saving) 

Spatio-temporal 

N/A 

N/A 

N/A 

N/A 

*note:  number  of  samples  is  the  number  of  wells  (spatial)  or  the  number  of  samples  per  year  (spatiotemporal) 

Aquifers:  Shallow,  Intermediate,  and  Deep  (evaluated  separately) 

COCs  Evaluated:  TCE  and  RDX  (evaluated  simultaneously,  errors  for  each  COC  added  together) 
Model:  Kriging  interpolation  with  quantile  data  transformation 


Tables  5-2  to  5-4  clearly  indicate  that,  for  the  two  sites  where  spatiotemporal  analysis  was 
performed,  the  potential  savings  achieved  with  spatial  optimization  was  far  greater  than  the 
potential  savings  achieved  with  spatiotemporal  optimization  (no  spatiotemporal  analysis  was 
performed  for  the  NOP  site).  Potential  savings  from  spatial  analysis,  calculated  based  on  number 
of  wells  eliminated,  ranged  from  approximately  10%  to  approximately  67%.  A  value  of 
approximately  35%  appears  to  be  representative.  The  ultimate  savings  would  be  even  greater  if 
some  reduction  in  sampling  frequency  was  implemented  at  some  of  the  remaining  wells  (using 
some  sort  of  rule  to  fill  in  values  for  wells  not  sampled  in  specific  events  to  make  plume  maps, 
estimate  plume  mass,  etc.).  The  potential  savings  from  the  spatiotemporal  analysis,  based  on 
number  of  samples  eliminated  per  year,  only  ranges  from  approximately  4%  to  approximately 
1 7%  at  the  two  sites  where  spatiotemporal  analysis  was  performed.  An  underlying  conceptual 
reason  for  this  is  discussed  in  Section  6.1.4  of  the  Technical  Report. 
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Figure  5-3  is  an  example  of  plot  provided  by  the  Data  Tracker  module  of  the  software  illustrating 
an  “out-of-bounds”  value  in  the  most  recent  sampling  event  that  might  be  of  concern.  In  this 
instance  there  are  only  six  background  data  values  with  a  statistically  significant  decreasing 
trend,  so  DT  selects  exponentially  decreasing  prediction  bounds;  these  are  fairly  wide  due  to  the 
limited  amount  of  data  available.  In  this  case,  the  analyst  would  examine  the  plot  and  the  data 
values  and  most  likely  decide  that  the  TCE  levels  in  MW-23B  were  by  now  essentially  around  or 
less  than  the  usual  reporting  limit,  and  that  the  only  “action”  appropriate  would  be  to  delete  the 
early  high  values  from  the  background  data  set  as  soon  as  the  requisite  minimum  number  of 
relevant  new  background  data  values  became  available. 


Figure  5-3.  Example  Data  Tracker  Plot  With  “Out-of-Bounds”  Value 
That  is  a  Potential  Concern 


MW-23B  -  TCE  (Out  Of  Bounds) 


Date 


■  Upper  Bound 

■  Concentration 

■  Lower  Bound 


A  summary  of  the  amount  of  time  it  takes  to  apply  the  software  is  presented  in  Table  5-5.  This  is 
primarily  an  indication  of  the  computation  time,  though  the  data  preparation  task  is  primarily 
associated  with  manual  labor.  The  computation  time  provided  for  Model  Builder,  spatial 
optimization,  and  spatiotemporal  optimization  are  for  each  problem  that  was  simulated  (e.g., 
each  aquifer  was  a  separate  problem  for  the  demonstration  sites).  Also,  additional  time  beyond 
the  computation  time  is  required  to  interpret  results. 
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Table  5-5.  General  Summary  of  Time  Required  to  Apply  the  Software 


Task 

Time* 

Comments 

Data  Cleanup,  Screening, 
and  Formatting 

Several  days 
(labor) 

Similar  effort  is  needed  to  apply  any  LTMO  software; 
effort  primarily  manual  labor 

Model  Builder 

Minutes** 

More  time  for  kriging  and  higher  spatial  resolution 

Spatial  Optimization 

Minutes  to  Hours** 

Computation  time  increases  with  “population  size”  and 
“generation  number”  for  GA 

Spatio-Temporal 

Optimization 

Hours  to  Days** 

Computation  time  increases  with  “population  size”  and 
“generation  number”  for  GA 

Data  Tracker 

Minutes  to  hours 
(data  preparation 
and  interpretation) 

Preparation  of  data  and  plotting  results  requires  most  of 
the  time 

*for  tasks  where  computation  time  is  indicated,  additional  time  is  required  for  interpretation  of  results 
**computation  time  per  problem  (e.g.,  per  aquifer) 


A  more  detailed  summary  of  eomputation  time  required  for  the  Model  Builder  and  redundaney 
evaluation  at  the  different  demonstration  sites  is  presented  in  Table  5-6. 


Table  5-6.  Summary  of  Computation  Time  Versus  Number  of  Wells 
_ (Kriging  with  Quantile  Transformation) _ 


Site  Name 

Aquifer 

#of 
samples 
per  year 

Computation  Time* 

Model  Builder 
(kriging  interpolation) 

Sampling  Optimizer 

SO 

STO 

Camp  Allen 
Site 

Shallow 

42  for  SO 

2-4  hours** 

30-60  minutes 

N/A 

Deep 

31  for  SO 

21  for  STO 

2-4  hours** 

30-60  minutes 

5-6  hours 

GAFB 

Upper 

55  for  SO 
108  for  STO 

20  minutes 

10-30  minutes 

several  days 

NOP 

Shallow 

81  for  SO 

~5  minutes 

15-50  minutes 

N/A 

Intermediate 

84  for  SO 

~5  minutes 

15-50  minutes 

N/A 

Deep 

56  for  SO 

~5  minutes 

15-50  minutes 

N/A 

N/A  =  Not  Analyzed;  SO  =  Spatial  Optimization;  STO  =  spatiotemporal  optimization 
*Pentium  4,  3.2  GHz 

**the programming  was  changed  to  increase  speeds  after  the  Camp  Allen  site  was  evaluated 


The  eomputation  times  in  Table  5-6  are  based  on  kriging  and  quantile  transformation,  whieh  was 
determined  to  be  the  preferred  model  with  respeet  to  the  representation  of  the  plume.  The  ranges 
in  eomputation  time  are  due  to  some  of  the  other  parameters  sueh  as  number  of  vertical  slices 
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(resolution)  and  the  GA  parameters  sueh  as  population  size.  It  is  also  noted  that  the 
programming  of  Model  Builder  was  improved  after  the  evaluation  of  the  Camp  Allen  site, 
leading  to  faster  eomputation  speeds  for  the  other  two  sites. 


5.5  SAMPLING  METHODS 

No  samples  were  collected  by  the  ESTCP  project  team  as  part  of  this  project.  The  data  that  were 
utilized  were  from  sampling  results  previously  obtained  by  the  demonstration  sites  under  their 
site-specific  sampling  plans. 


5.6  SAMPLING  RESULTS 

Again,  no  samples  were  collected  by  the  ESTCP  project  team  as  part  of  this  project.  The  data 
that  were  utilized  were  from  sampling  results  previously  obtained  by  the  demonstration  sites 
under  their  site-specific  sampling  plans. 
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6.0  PERFORMANCE  ASSESSMENT 


6.1  QUALITATIVE  PERFORMANCE  OBJECTIVES 

6.1.1  User  Functionality 

The  expected  performance  metric  is  that  the  Summit  Software  has  an  acceptable  learning  curve 
(e.g.,  1-2  days)  that  will  not  discourage  prospective  users  and  will  allow  users  to  achieve  the 
intended  objectives.  For  the  first  site,  a  mid-level  analyst  was  provided  a  one-day  training 
session  by  the  software  developer  prior  to  using  the  software.  For  the  second  and  third 
demonstration  sites,  a  different  mid-level  analyst  was  used,  and  that  person  relied  only  on  the 
software  documentation  plus  phone  support  by  the  software  developer.  Based  on  the  application 
of  the  software  at  all  three  demonstration  sites,  this  performance  objective  was  met. 


6,1.2  Software  Reliability 

The  expected  performance  metric  is  that  the  Summit  Software  has  no  significant  errors  or  bugs 
remaining  by  the  end  of  this  project.  Any  bugs  that  were  identified  were  reported  to  the  software 
developer,  who  then  fixed  the  problems  such  that  the  performance  objective  was  met. 


6,1,3  Model  Builder  Performance 

The  expected  performance  metric  is  that  Model  Builder  provides  a  model  of  spatial  and/or 
temporal  variation  for  each  primary  COC  at  each  site  that  is  adequate  given  the  available  data. 
Model  Builder  allows  for  two  different  interpolation  techniques  and  three  different  data 
transformation  techniques,  for  a  total  of  six  combinations.  The  reasonableness  of  all  six 
combinations  was  evaluated  at  each  of  the  three  sites.  In  each  case,  the  analyst  reviewed  the 
plume  map  generated  by  the  software  for  each  combination  of  interpolation  and  transformation, 
and  qualitatively  assessed  the  reasonableness  of  the  plume  map. 

The  qualitative  performance  objective  for  Model  Builder  was  met  with  respect  to  modeling  of 
spatial  variation.  With  respect  to  spatial  variation,  the  analyst  established  that  the  combination  of 
kriging  with  quantile  transformation  qualitatively  provided  the  most  reasonable  representation  of 
the  data,  which  is  consistent  with  the  recommendations  in  the  software  manual.  In  general,  using 
no  data  transformation  resulted  in  unacceptably  poor  representation  of  the  data. 

The  qualitative  performance  objective  for  Model  Builder  was  only  partially  met  with  respect  to 
modeling  of  temporal  variation.  Model  Builder  addresses  temporal  changes  in  concentration  in 
support  of  several  aspects  of  software  functionality,  such  as: 

•  Spatiotemporal  optimization  for  reducing  sampling  redundancy  (in  both  space  and  time) 
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•  Calculation  of  relative  mass  or  mass  flux  in  different  sampling  events,  based  on 
interpolation  of  sampling  results  in  eaeh  event 

It  beeame  apparent  during  implementation  of  the  software  at  the  demonstration  sites  that,  for 
eaeh  of  these  software  funetions  listed  above.  Model  Builder  performs  a  series  of  spatial 
interpolations  over  time,  but  does  not  perform  any  interpolation  with  respeet  to  time.  This  is 
easily  explained  with  a  eoneeptual  example,  illustrated  in  Figure  6-1. 


Figure  6-1  Coneeptual  Example  to  Illustrate  Temporal  Interpolation  Issue 

Sampling  Event  1  Sampling  Event  2  Sampling  Event  3 


100 

100 

• 

• 

Not  sampled 

• 

100 

100 

• 

• 

In  Figure  6-1,  the  only  differenee  between  the  three  sampling  events  is  that  the  middle  point  is 
not  sampled  in  Sampling  Event  2.  The  software  only  performs  spatial  interpolation  within  eaeh 
event.  Therefore,  in  Sampling  Event  #2,  the  software  will  interpolate  values  inside  the  four 
aetual  samples  based  only  on  those  four  values.  Sinee  those  values  are  eaeh  100,  presumably  the 
software  will  interpolate  a  value  of  100  at  eaeh  loeation  inside  the  four  aetual  values.  However, 
it  seems  unlikely  that  an  environmental  seientist  provided  with  the  data  illustrated  in  Eigure  6-1 
would  eonsider  that  the  most  reasonable  approaeh.  Rather,  a  temporal  interpolation  at  the 
loeation  not  sampled  in  Sampling  Event  2  based  on  the  other  two  events  would  appear  to  be 
more  appropriate,  or  perhaps  some  eombination  of  temporal  and  spatial  interpolation.  More 
speeifieally,  an  environmental  seientist  would  likely  eonelude  that  that  the  value  at  the  loeation 
not  sampled  in  Sampling  Event  2  is  probably  elose  to  500,  and  then  perform  subsequent  spatial 
interpolation  in  the  remaining  area  aeeordingly. 

Beeause  of  this  issue,  the  GeoTrans  analyst  eoneluded  that  the  qualitative  performanee  objeetive 
regarding  Model  Builder  is  only  partially  met  with  respeet  to  temporal  variation.  Temporal 
variation  is  modeled  adequately  only  if  the  sampling  loeations  are  eonsistent  between  sampling 
events,  or  if  the  user  fills  in  missing  values  in  speeifie  events  manually  prior  to  use  of  the 
software,  based  on  eonsiderations  ineluding  temporal  interpolation.  Otherwise,  the 
representation  of  temporal  variability  may  not  be  adequately  modeled  within  Model  Builder. 
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6.1,4  Sampling  Optimizer  Performance 

The  expected  performance  metric  is  that  Sampling  Optimizer  provides  reasonable  trade-off 
curves  allowing  site  personnel  and  other  professionals  to  easily  identify  optimal  monitoring 
program  choices.  Based  on  the  application  of  the  software  at  all  three  demonstration  sites,  the 
qualitative  performance  objective  for  Sampling  Optimizer  was  met  for  spatial  optimization,  but 
was  only  partially  met  for  spatiotemporal  optimization.  In  the  case  of  spatiotemporal 
optimization,  the  GeoTrans  analyst  noted  that  the  results  seemed  overly  conservative  because  the 
software  tended  not  to  eliminate  wells  entirely  from  the  recommended  sampling  plans.  This  was 
potentially  not  reasonable  given  the  corresponding  spatial  optimization  results  which  suggested 
that  many  wells  could  be  eliminated  with  acceptable  levels  of  error.  Details  regarding  specific 
results  for  demonstration  sites  that  pertain  to  this  issue  were  presented  in  the  Technical  Report. 
However,  further  explanation  of  an  underlying  issue  associated  with  spatiotemporal  optimization 
is  provided  below. 

Spatiotemporal  results  are  inherently  more  conservative  than  the  spatial  results  alone,  because 
the  optimization  problem  has  additional  constraints.  However,  the  fact  that  the  software  does  not 
perform  temporal  interpolation  of  data  provides  an  additional  component  of  conservatism. 

The  reason  why  the  spatiotemporal  optimization  can  be  overly  biased  towards  not  eliminating 
wells  is  illustrated  using  a  conceptual  example  presented  in  Figure  6-2.  This  figure  illustrates 
data  values  from  three  sampling  events. 

Figure  6-2.  Conceptual  Example  to  Illustrate  Spatiotemporal  Optimization  Issue 


Sampling  Event  1  Sampling  Event  2  Sampling  Event  3 


100 

100 

• 

51  49 

A  B 

• 

100 

100 

• 

• 

100 

100 

• 

Not  Sampled  49 

A  B 

• 

100 

100 

• 

• 

100 

100 

• 

• 

51  Not  Sampled 

A**B 

100 

100 

• 

• 

On  Figure  6-2.  note  the  locations  labeled  “A”  and  “B”  in  the  middle  of  each  event.  In  Sampling 
Event  1,  both  of  those  locations  are  sampled.  In  Sampling  Event  2  location  “A”  is  not  sampled, 
and  in  Sampling  Event  3  location  “B”  is  not  sampled.  It  appears  based  on  inspection  that 
locations  “A”  and  “B”  appear  to  be  redundant.  It  is  likely  that  the  missing  value  at  “A”  in 
Sampling  Event  2  is  close  to  51,  and  likely  that  the  missing  value  at  “B”  in  Sampling  Event  3  is 
close  to  49.  Eurthermore,  it  would  not  cause  significant  interpolation  error  if  either  “A”  or  “B” 
was  permanently  eliminated,  as  long  as  a  value  at  the  other  well  was  available. 

However,  the  software  would  likely  not  reach  the  same  conclusion.  Spatiotemporal 
optimization,  as  implemented  in  the  software,  provides  a  tradeoff  curve  of  sampling  cost  versus 
sampling  error.  In  a  spatiotemporal  optimization,  the  error  calculated  by  the  software  for  each 
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potential  sampling  plan  is  the  maximum  error  at  any  sampling  loeation  in  any  sampling  event, 
ealeulated  at  loeations/times  where  a  sample  is  removed,  based  on  spatial  interpolation  using  the 
remainins  samples  in  that  event.  As  diseussed  with  respeet  to  Model  Builder,  no  temporal 
interpolation  is  performed  within  the  software  to  address  the  missing  values  as  part  of  these 
ealeulations.  Thus,  in  the  example  in  Figure  6-2,  the  software  would  generally  not  eliminate 
loeation  “A”  beeause  it  would  lead  to  a  large  error  in  Sampling  Event  3  at  loeation  “A”  (beeause 
no  value  would  be  assumed  at  loeation  “B”  during  that  event).  Similarly,  the  software  would  not 
eliminate  loeation  “B”  beeause  it  would  lead  to  a  large  error  in  Sampling  Event  2  at  loeation  “B” 
(beeause  no  value  would  be  assumed  at  loeation  “A”  during  that  event).  Thus,  the  ultimate  result 
from  the  software  would  likely  inelude  eontinued  sampling  at  both  wells  at  some  frequeney, 
rather  than  eliminating  one  of  the  two  wells. 

This  underlying  issue  will  only  be  signifieant  when  well  loeations  are  not  eonsistent  for  eaeh 
sampling  event.  However,  it  is  very  often  the  ease  that  at  least  some  sampling  events  will  have 
different  sampling  loeations  than  others  (due  to  different  sampling  frequeneies,  new  wells, 
abandoned  wells,  and  wells  that  eannot  be  sampled  in  speeifie  events  due  to  logisties).  Beeause 
the  software  does  not  provide  for  temporal  interpolation,  those  missing  values  are  not  adequately 
represented  within  the  spatiotemporal  optimization  proeess.  The  assignment  of  these  missing 
values  ean  be  assigned  external  to  the  software,  but  that  is  potentially  a  labor-intensive  proeess. 
Moreover,  assigning  values  to  missing  data  using  any  form  of  interpolation  involves  making 
subjeetive  judgments  about  the  temporal  smoothness  of  the  data;  as  a  result,  a  temporal 
optimization  will  be  evaluating  some  eombination  of  the  aetual  temporal  redundaney  and  the 
temporal  redundaney  of  the  subjeetive  judgments,  rather  than  evaluating  only  the  aetual  temporal 
redundaney.  It  seems  likely  that  this  issue  would  be  problematie  to  any  approaeh  to  temporal 
optimization  that  one  might  attempt  to  use  with  datasets  eontaining  numerous  missing  values. 

Therefore,  in  summary,  the  software  does  produee  reasonable  tradeoff  eurves  for  spatial 
optimization,  and  in  some  eases  where  sampling  loeations  are  not  eonsistently  sampled,  the 
tradeoff  eurves  for  spatiotemporal  optimization  may  be  overly  eonservative  with  respeet  to 
elimination  of  wells. 


6.1,5  Data  Tracker  Performance 

The  expeeted  performanee  metrie  is  that  Data  Traeker  enables  the  easy  ineorporation  of  site- 
speeifie  monitoring  and  remediation  expeetations  and  data  objeetives  along  with  historieal  data. 
The  qualitative  performanee  objeetive  for  Data  Traeker  was  partially  met.  Data  Traeker  was 
found  to  be  easy  to  use.  Data  Traeker  does  allow  traeking  of  reeently  eolleeted  eoneentration 
data  against  statistieally  eomputed  bounds  that  are  ealeulated  from  the  historieal  (i.e., 
baekground)  data.  The  software  indieates  whieh  of  the  reeently  eolleeted  data  values  are  “out- 
of-bounds”  relative  to  expeetations  that  are  ealeulated  from  the  baekground  values.  However  the 
software  does  not  appear  to  traek  any  quantity  with  respeet  to  “remediation  expeetations”  or 
“site-speeifie  objeetives”  as  stated  in  the  performanee  objeetives.  With  respeet  to  eoneentrations, 
the  software  does  not  indieate  whieh  values  are  above  or  below  remediation  goals.  With  respeet 
to  other  parameters  that  ean  be  traeked  by  the  software,  sueh  as  mass  and  mass  flux,  the  software 
will  provide  a  table  of  historieal  versus  eurrent  values,  but  it  does  not  eompare  these  to  any  site- 
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specific  remediation  goals  for  those  parameters.  Therefore,  the  software  does  effectively  track 
recently  collected  data  versus  historical  data,  and  does  allow  quantities  such  as  relative  plume 
mass  and  mass  flux  across  a  site  boundary  to  be  tracked,  but  does  not  specifically  address  “site- 
specific  remediation  expectations”  within  the  software.  In  fairness,  one  should  note  that  tracking 
progress  toward  site-specific  remediation  goals  did  not  arise  in  discussions  of  objectives  with  site 
personnel  at  any  of  the  three  demonstration  sites. 


6,1,6  Regulatory  Acceptance 

The  expected  performance  metric  is  that  the  results  of  this  ESTCP  dem/val  will  be  persuasive  to 
regulatory  personnel.  Site  personnel  for  the  demonstration  sites  indicated  that  the  types  or 
tradeoff  curves  produced  by  the  software  for  evaluating  redundancy  (based  on  mathematical 
optimization),  in  conjunction  with  the  comparison  of  plume  visualizations  with  and  without 
redundant  data  that  are  produced  by  the  software,  would  be  convincing. 


6,1,7  Comparison  with  MAROS  (secondary) 

The  expected  performance  metric  is  that  the  Summit  Software  will  be  found  to  be  at  most 
modestly  more  difficult  to  learn  to  use  compared  to  MAROS.  This  was  evaluated  in  this  project 
by  applying  both  software  products  at  the  first  site,  conducted  by  the  same  analyst,  with 
equivalent  training  in  both  software  products.  Based  on  this  effort,  the  performance  objective 
was  met.  The  analysts  reported  that  both  software  products  were  equally  easy  to  learn  and 
implement.  In  the  case  of  applying  MAROS,  there  was  some  initial  confusion  about  how  to 
utilize  some  of  the  input  parameters  (current  plume  length,  distance  from  source,  etc.)  for  the 
specific  problem  being  solved,  but  those  were  easily  addressed  by  corresponding  with  the 
software  developer.  Similarly,  there  were  some  minor  questions  about  what  values  to  assign  for 
some  of  the  Summit  Software  parameters,  and  those  were  easily  addressed  by  corresponding 
with  the  software  developer  (some  of  those  issues  were  subsequently  addressed  by  improvements 
made  to  the  software  and  added  to  the  User’s  Manual  during  the  remainder  of  the  project).  More 
detail  regarding  the  comparison  between  the  Summit  Software  and  MAROS  is  presented  in 
Appendix  F  of  the  Technical  Report. 


6,2  QUANTITATIVE  PERFORMANCE  OBJECTIVES 
6,2,1  Model  Builder  Performance 

The  expected  performance  metric  is  that  Model  Builder  provides  a  model  of  spatial  and/or 
temporal  variation  for  each  primary  constituent  of  concern  at  each  site  that  is  adequate  given  the 
available  data.  This  was  evaluated  in  this  project  by  testing  the  various  Model  Builder  options  at 
the  three  demonstration  sites,  and  making  visual  comparisons.  It  is  not  clear  that  there  is  a  more 
quantitative  manner  to  evaluate  this  objective.  Thus,  the  conclusions  are  the  same  as  stated  for 
the  qualitative  performance  objective  for  Model  Builder  (see  Section  6.1). 
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6,2.2  Sampling  Optimizer  Performance 

The  expected  performance  metric  is  that  the  optimized  programs  identified  by  Sampling 
Optimizer  in  fact  permit  cost  reductions  with  acceptable  losses  of  information,  if  appropriate,  as 
anticipated  for  the  large  majority  of  DoD  sites.  This  performance  objective  was  met.  At  each  of 
the  three  demonstration  sites,  sampling  plans  were  selected  from  the  tradeoff  curves  (sampling 
cost  versus  error)  with  substantially  lower  cost  that  the  analyst  felt  had  acceptable  error. 
Furthermore,  at  each  of  the  three  sites  validation  was  performed  on  a  subsequent  data  set  to 
determine  if  the  errors  that  might  result  from  the  reduced  sampling  in  the  recommended  plans 
were  acceptable,  and  in  each  case  it  was  confirmed  that  a  plan  recommended  by  the  software 
with  significantly  reduced  sampling  resulted  in  acceptable  error  when  applied  to  the  reserved 
dataset.  The  validation  exercise  provided  confidence  in  the  results  provided  by  SO. 


6.2,3  Data  Tracker  Performance 

The  expected  performance  metric  is  that  Data  Tracker  responds  appropriately  to  artificially 
induced  anomalies  of  interest  for  the  particular  site.  This  was  evaluated  in  this  project  by  having 
EnviroStat  create  artificial  anomalies  in  consultation  with  site  and  other  personnel  at  each  of  the 
three  demonstration  sites.  Those  data  were  provided  to  the  software  analyst  from  GeoTrans  who 
used  Data  Tracker  in  a  single-blind  evaluation  (i.e.,  the  GeoTrans  analyst  was  provided  six 
variations  of  the  actual  “current  data”  with  no  external  clues  as  to  which  was  correct  and  which 
had  artificial  anomalies).  EnviroStat  subsequently  evaluated  the  extent  to  which  the  GeoTrans 
analyst  was  able  to  detect  these  artificial  anomalies  with  the  software,  and  provided  a  summary 
write-up  for  each  site.  Based  on  those  efforts,  this  performance  objective  was  met.  The  vast 
majority  of  artificial  anomalies  were  detected  (details  provided  in  the  Technical  Report). 


6,2,4  Comparison  with  MAROS 

The  expected  performance  metric  is  that:  1)  the  Summit  Software  will  be  found  to  be  at  most 
modestly  more  difficult  to  learn  to  use,  consistent  with  being  much  more  flexible  in 
incorporating  site-specific  monitoring  objectives;  and  2)  if  both  products  are  able  to  accept  the 
same  goals  and  constraints,  results  will  be  similar  but  slightly  different  due  to  small  differences 
due  to  different  optimization  methodologies.  There  were  no  pre-conceptions  regarding  the 
comparisons  of  optimization  recommendations  to  be  expected  from  the  two  software  products. 

With  regard  to  ease  of  use,  GeoTrans  personnel  received  training  for  both  the  Summit  and 
MAROS  tools  by  the  software  developers.  The  Summit  training  was  a  little  more  than  a  half¬ 
day,  and  the  MAROS  training  was  approximately  a  half-day  hands-on  training  that  was  part  of  a 
two-day  ETMO  conference.  Both  software  products  were  similarly  easy  to  install  and  could  be 
learned  and  used  by  people  with  similar  training  and  qualifications.  Both  Elser’s  Guides  are 
comprehensive  and  clearly  presented.  Similarly,  to  get  historical  site  data  into  the  required 
format  for  input  to  the  software  was  no  more  significant  for  one  software  versus  the  other 
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software;  the  user  simply  needs  to  follow  the  instructions  regarding  the  input  structure  and 
requirements  for  that  software  product.  These  modifications  took  on  the  order  of  minutes  to 
several  hours  for  each  software  product  for  someone  experienced  with  performing  such 
operations  in  MS  Excel  or  MS  Access. 

With  regard  to  results,  comparisons  of  specific  recommendations  provided  by  each  software 
package  are  difficult,  as  explained  in  detail  in  Appendix  F  of  the  Technical  Report.  Key 
comparison  observations  that  can  be  made  include  the  following: 

•  The  primary  advantage  of  the  Summit  Software  is  that  the  redundancy  evaluation  is 
based  on  mathematical  optimization  which  allows  sampling  redundancy  to  be  evaluated 
on  a  system-wide  basis  (e.g.,  best  solution  if  one  location  is  removed,  if  two  locations 
are  removed,  if  three  locations  are  removed,  etc.).  A  key  benefit  of  this  approach  is  that 
it  allows  the  tradeoff  between  the  number  of  samples  and  the  accuracy  of  the  resulting 
plume  interpolation  to  be  assessed.  This  is  a  significant  improvement  over  the  approach 
for  evaluating  data  redundancy  utilized  in  MAROS,  which  is  not  based  on  mathematical 
optimization.  In  MAROS,  individual  well  locations  are  evaluated  for  redundancy  based 
on  impacts  of  removing  that  well  alone;  consequently,  the  impact  of  removing  groups  of 
wells  cannot  be  assessed  and  the  aforementioned  tradeoff  cannot  be  evaluated. 

•  The  Summit  Software  approach  to  data  redundancy  evaluation  provides  plume 
visualizations  for  the  baseline  plan  (i.e.,  all  samples)  versus  improved  plans  (i.e.,  reduced 
numbers  of  samples)  within  the  software.  These  comparative  visualizations  are  quite 
effective  for  communication  with  stakeholders  and  regulators.  However,  these  maps  can 
only  be  exported  as  image  files,  thus,  it  is  difficult  to  directly  import  these  image  files 
into  other  software  packages  such  as  Surfer,  ArcGIS,  and  AutoCAD.  MAROS  does  not 
include  such  plume  visualizations. 

•  The  Summit  Software  has  a  Data  Tracker  module  that  indicates  if  new  data  are  “in¬ 
bounds”  or  “out-of-bounds”  relative  to  expectations,  based  on  previous  data  at  that  well. 
This  functionality  is  useful  but  is  not  present  in  MAROS.  MAROS  indicates  if  the 
concentration  trend  at  a  well  is  increasing,  decreasing,  or  stable.  This  functionality  is 
also  useful,  but  is  not  present  in  the  Summit  Software. 

•  Both  software  products  suffer  from  some  similar  limitations.  With  respect  to  areas  of 
uncertainty,  neither  software  package  provides  specific  recommendations  (i.e.,  number 
of  new  wells  and/or  locations  of  new  wells)  to  reduce  the  uncertainty.  With  respect  to 
mass  calculations,  neither  software  performs  temporal  interpolation  or  extrapolation  to 
fill  in  missing  values  in  events  where  specific  wells  are  not  sampled.  As  a  result,  mass 
or  mass  flux  results  will  have  higher  variability  and  uncertainty  for  events  with  fewer 
samples,  different  spatial  distribution  of  samples,  and/or  for  events  where  key  wells 
(e.g.,  wells  with  high  concentrations)  are  not  sampled. 

Additional  comparison  with  MAROS  is  planned  using  data  from  the  NOP  site,  but  that 
comparison  will  be  performed  outside  the  scope  of  this  ESTOP  project. 
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7.0  COST  ASSESSMENT 


7.1  COST  MODEL 

The  software  is  free  for  use  at  DoD  sites.  Furthermore,  sinee  the  software  runs  on  standard 
desktop  computers  no  capital  purchases  are  required.  Therefore,  the  cost  of  implementation  is 
the  estimated  cost  of  applying  the  software  at  a  typical  site,  and  perhaps  some  minor  training 
costs  for  initial  use.  For  the  demonstration  project,  approximately  $60,000  per  site  was  allocated 
for  testing  the  software.  However,  this  is  far  in  excess  of  what  would  be  required  for  a  typical 
site.  This  is  because  many  potential  variations  were  addressed  during  this  project  to  allow  for 
robust  testing  of  the  software.  In  Table  7-1,  estimates  are  provided  for  applying  the  software  at  a 
typical  DoD  site  assuming  that  the  redundancy  evaluation  will  be  performed  spatially  rather  than 
spatiotemporally. 


Table  7-1.  Estimated  Costs  to  Apply  the  Software  at  a  Typical  DoD  Site 


Cost  Element 

Estimated  Level  of  Effort 

Estimated  Cost 

Start-Up 

Software  Cost 

Free 

$  0 

Software  download 

1  hr  @$  100/hr 

$  100 

Training/leaming 

16  hrs  @$  100/hr 

$1,600 

Subtotal 

- >$1,700 

Redundancy  Evaluation  fPeriodicl 

Per  Site'. 

Formulation 

Lump  sum 

$5,000 

Data  Prep 

24  hrs  $  100/hr 

$2,400 

Import  Data  Into  Software 

2  hrs  @$  100/hr 

$  200 

subtotal 

- >$7,600 

Per  Plume  Evaluated*'. 

Model  Builder 

2  hrs  @  100/hr 

$  200 

Optimization  ** 

24  hrs  @  100/hr 

$2,400 

Interpret  Results  and  Write  Up 

20  hrs  @  100/hr 

$2,000 

subtotal 

- >$4,600  (per  plume) 

Data  Tracker 

First  Time: 

Develop  Initial  Background  Data  File 

(Part  of  Data  Prep  listed  above) 

$  0 

Each  Year: 

Evaluate  Need  to  Update  Background 

16  hrs@  100/hr 

$1,600 

subtotal 

- >$1,600  (per  year) 

Each  Event: 

Create  CSV  File  for  New  Data 

2  hrs  @  100/hr 

$  200 

Import  Data  and  Run  DT 

1  hrs  @  100/hr 

$  100 

Export  Charts,  Print  Charts,  Interpret 

5  hrs  @  100/hr 

$  500 

subtotal 

- >$800  (per  eveut) 

*each  plume  may  consist  of  multiple  primary  COCs,  but  each  aquifer  or  aquifer  horizon  where  the  plume  is 
represented  with  a  different  map  would  be  treated  as  separate  plume 


**assumes  several  variations  will  be  attempted  such  as  changing  the  Model  Builder  algorithm  or  the  list  of 
excluded  wells 
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7.2  COST  DRIVERS 


The  cost  estimates  provided  in  Table  7-1  are  rough  estimates  based  on  the  testing  performed  as 
part  of  this  demonstration  project.  Some  cost  drivers  that  would  potentially  impact  the  cost  of 
applying  the  software  are  provided  below: 

•  formulation  task  will  depend  on  number  of  people  and  need  for  additional  meetings 

•  data  preparation  cost  will  depend  on  the  quality  of  the  site  data 

•  redundancy  evaluation  costs  depend  on  the  number  of  plumes 

•  spatiotemporal  optimization  requires  much  more  computation  time  than  spatial 
optimization 

The  items  above  are  discussed  in  detail  in  the  Technical  Report.  Also,  the  computation  time  is 
somewhat  impacted  by  the  number  of  wells  and  the  model  type  (kriging  versus  inverse  distance 
weighting),  but  these  variations  should  not  significantly  impact  the  costs  estimated  in  Table  7-1. 
It  is  also  noted  that  the  labor  cost  estimates  in  Table  7-1  are  approximations  that  may  differ  from 
site  to  site. 


7.3  COST  ANALYSIS 

A  cost-benefit  analysis  for  applying  this  LTMO  software  must  account  for  the  costs  of  applying 
the  technology  and  the  cost  savings  likely  to  be  realized.  The  estimated  costs  of  applying  the 
technology  were  presented  in  Table  7-1 .  The  costs  savings  will  result  from  reduced  labor  and 
analysis  associated  with  the  elimination  of  some  sampling. 

The  actual  costs  and  savings  are  subject  to  many  site-specific  factors  such  as  the  number  of 
aquifers,  the  number  of  wells,  the  cost  of  sampling,  the  cost  of  laboratory  analysis,  and  many 
other  factors.  Since  these  factors  vary  from  site  to  site,  examples  are  provided  below  to  illustrate 
how  the  costs  and  savings  can  be  evaluated. 

For  the  first  scenario,  the  following  assumptions  are  made: 

•  Evaluate  a  10-year  monitoring  horizon 

•  The  LTMO  costs  are  based  on  the  values  estimated  in  Table  7-1 

•  There  is  only  one  aquifer,  and  the  redundancy  evaluation  is  performed  on  one 
comprehensive  plume 

•  The  redundancy  evaluation  is  performed  once  at  the  beginning  of  the  10-year  period,  and 
again  in  year  6  of  the  10-year  period.  To  support  the  second  periodic  evaluation,  the  full 
set  of  monitoring  wells  is  sampled  for  one  of  the  two  events  in  year  5  (i.e.,  year  five  only 
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has  half  the  savings  associated  with  reduced  amount  of  wells  sampled).  For  simplicity, 
we  assume  the  same  level  of  sampling  reduction  after  this  second  round  of  optimization 
versus  the  original  baseline  number  of  wells. 

•  The  wells  are  sampled  twice  per  year 

•  There  are  60  total  samples  per  sampling  event  in  the  current  monitoring  plan  (i.e.,  120 
samples  per  year) 

•  The  cost  of  collecting  a  sample,  plus  the  laboratory  cost  for  analysis,  is  $800 

•  The  spatial  LTMO  analysis  eliminates  35%  of  the  sampling  locations  (representative 
results  for  spatial  optimization  for  the  three  demonstration  sites,  as  described  in  Section  5 
and  presented  on  Tables  5-2  to  5-4) 

•  Future  costs  are  discounted  to  present  day  dollars  using  a  10-year  discount  rate  of  2.6% 
as  per  0MB  (www.whitehouse.gov/omb/circulars/a094/a94_appx-c.html). 

Other  minor  savings  might  occur  because  fewer  duplicate  samples  and  QA/QC  samples  (e.g.,  trip 
blanks  and  field  blanks)  may  be  required,  but  those  details  have  not  been  included.  Also,  the 
additional  costs  of  evaluating  the  current  data  for  unexpected  values  without  using  data  tracker 
are  hard  to  quantify  and  are  not  included. 

The  cost  benefit  analysis  for  this  scenario  is  summarized  on  Table  7-2. 


Table  7-2.  Cost-Benefit  Analysis,  Scenario  1 


Scenario  1 

Assumotions 

1 

#  of  Plumes 

5 

Frequncy  of  Periodic  Evaluation  (in  year) 

2 

Sampling  Frequency  (per  year) 

- 

— 

60 

Total  #  of  Samples  per  Event  (all  plumes  combined) 

$800 

Cost  per  Sample  (labor  plus  analysis) 

35% 

Savings  for  Redundancy  Analysis 

- 

260% 

Discount  Rate  (OMB  10  year  value) 

10 

Monitoring  Time  Frame  (In  years) 

Year 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Total 

1 

$1,700 

$1,700 

Periodic  Evaluation 

$24,400 

Data  T  racket 

$1,600 

$3,200 

$3,200 

$3,200 

$3,200 

$3,200 

$3,200 

$3,200 

$3,200 

$3,200 

1 

$30,400 

siibtolat  (mn-iUscountedh  > 

$15,500 

$3,200 

$3,200 

$3,200 

$3,200 

$15,400 

$3,200 

$3,200 

$3,200 

$3,200 

$56,500 

stibtoUi!  (discoufile(J}'> 

$15,500 

$3,119 

$3,040 

$2,963 

$2,888 

$13,545 

$2,743 

$2,540 

1 

mm 

1 

■HUH 

imiiiiii 

$33,600 

$33,600 

$33,600 

$33,600 

$16,800 

$33,600 

$33,600 

$33,600 

$33,600 

$319,200 

stibtolal  (discoutiledh> 

$33,600 

$32,749 

$31,919 

$31,110 

$15,161 

$29,553 

$28,804 

$28,074 

$27,363 

$26,669 

$285,001 

net  benefit  (discounted)  — > 

$233,384 

Notes; 

Costs  are  based  on  estimates  in  Table  7-1 

Costs  for  start'Up  assumed  to  be  $1 ,700  for  training  and  learning 

Costs  for  Periodic  Evaluation  assumed  to  be  $7,600  per  site  plus  $4,600  per  plume  evaluated 

Costs  for  Data  T racket  analysis  assumed  to  be  $800  per  event  plus  $1 ,600  per  year  to  assess  need  to  update  background  data 

Potential  savings  from  reduction  in  number  of  trip  blanks,  field  blanks,  duplicate  samples,  etc.  not  included  [ 

Less  savings  assumed  in  year  5  •  assume  sampling  at  all  wells  in  one  of  the  two  events  to  use  for  periodic  redundancy  evaluation 

_ 
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On  Table  7-2  the  LTM  savings  for  the  optimized  sampling  plan  is  ealculated  as  follows: 

•  For  most  years,  the  savings  is  $33,600.  This  is  ealeulated  as: 

60  samples/event  *  2  events/year  *  $800/sample  *  35%  reduetion  in  samples 

•  For  year  5,  it  is  assumed  that  the  full  set  of  wells  is  sampled  in  one  of  the  two  events,  to 
provide  data  to  for  a  periodie  redundaney  evaluation,  so  the  savings  in  that  year  is  only 
half  the  amount  as  in  the  other  years. 

On  Table  7-2,  the  net  present  value  of  the  LTMO  eosts  is  approximately  $50,000  and  the  net 
present  value  of  the  LTM  savings  is  approximately  $285,000.  Thus,  the  net  savings  is  over 
$230,000  over  10  years.  This  relatively  modest  savings  is  beeause  this  seenario  ineludes  only 
one  aquifer  with  a  total  of  60  wells,  and  a  relatively  low  cost  of  $800  per  sample  for  sampling 
labor,  analysis,  and  validation. 

A  second  scenario  is  the  same  as  the  first  scenario  except  for  the  following: 

•  There  are  three  aquifers  instead  of  one,  each  with  60  wells,  which  has  the  following 
ramifications: 

o  The  redundancy  analysis  includes  three  plumes  rather  than  one  plume 

o  The  total  number  of  samples  per  event  in  the  current  monitoring  plan  is  180 
instead  of  60  (i.e.,  3  aquifers  rather  than  1  aquifer) 

•  The  cost  of  collecting  a  sample,  plus  the  laboratory  cost  for  analysis,  is  $1,400  rather  than 
$800  (perhaps  more  parameter  types  to  be  analyzed  and/or  more  difficult  sampling 
conditions) 

In  this  case  the  cost  of  the  LTMO  evaluations  are  slightly  higher  (since  there  are  three  aquifers 
rather  than  one  aquifer).  However,  the  estimated  LTM  savings  for  an  optimized  plan  are  much 
greater,  since  the  baseline  plan  has  many  more  wells  and  the  cost  per  sample  is  higher. 

The  cost  benefit  analysis  for  this  scenario  is  summarized  on  Table  7-3. 
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Table  7-3.  Cost-Benefit  Analysis,  Seenario  2 


Scenario  2 

Assumotions 

1 

3 

#  of  Plumes 

5 

Frequncy  of  Periodic  Evaluation  (in  year) 

2 

Sampling  Frequency  (per  year) 

180 

Total  U  of  Samples  per  Event  (all  plumes  combined) 

- 

SI, 400 

Cost  per  Sample  (labor  plus  analysis) 

35% 

Savings  for  Redundancy  Analysis 

- 

260% 

Discount  Rate  (OMB  10  year  value) 

10 

Monitoring  Time  Frame  (in  years) 

I 

Year 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Total 

Start-Up 

$1,700 

Periodic  Evaluation 

awgnm 

$42,800 

Data  T  racker 

$3,200 

$3,200 

$3,200 

53,200 

$3,200 

$3,200 

$3,200 

$3,200 

$3,200 

$30,400 

subtotal  tmn-iUsvoimted}- > 

$24,700 

$3,200 

$3,200 

$3,200 

$3,200 

$24,600 

$3,200 

$3,200 

$3,200 

$3,200 

$74,900 

subtotal  (tliscoutUe(l}'> 

$24,700 

$3,119 

$3,040 

$2,963 

$2,888 

$21,637 

$2,743 

$2,674 

$2,606 

$2,540 

LTM  Savings: 

Optimized  Samplinq  Plan 

$176,400 

$176,400 

$176,400 

$176,400 

$88,200 

$176,400 

$176,400 

$176,400 

$176,400 

$176,400 

subtotal  (discoutUetlh> 

$176,400 

$171,930 

$167,573 

$163,326 

$79,594 

$155,154 

$151,222 

$147,390 

$143,655 

$140,014 

net  benefit  (discounted)  — > 

$1,427,348 

Notes: 

Costs  are  based  on  estimates  in  Table  7-1 

* 

Costs  for  start-up  assumed  to  be  $1 ,700  for  training  and  learning 

Costs  for  Periodic  Evaluation  assumed  to  be  $7,600  per  site  plus  $4,600  per  plume  evaluated 

Costs  for  Data  T racker  analysis  assumed  to  be  S800  per  event  plus  SI  ,600  per  year  to  assess  need  to  update  background  data 

Potential  savings  from  reduction  in  number  of  trip  blanks,  field  blanks,  duplicate  samples,  etc.  not  included 

Less  savings  assumed  in  year  5  •  assume  sampling  at  all  \vells  in  one  of  the  two  events  to  use  for  periodic  redundancy  evaluation 

On  Table  7-3  the  LTM  savings  for  the  optimized  sampling  plan  is  ealeulated  as  follows: 

•  For  most  years,  the  savings  is  $176,400.  This  is  ealeulated  as: 

180  samples/event  *  2  events/year  *  $  1,400/sample  *  35%  reduetion  in  samples 

•  For  year  5,  it  is  assumed  that  the  full  set  of  wells  is  sampled  in  one  of  the  two  events,  to 
provide  data  to  for  a  periodie  redundaney  evaluation,  so  the  savings  in  that  year  is  only 
half  the  amount  as  in  the  other  years. 

As  summarized  on  Table  7-3,  the  net  present  value  of  the  LTMO  eosts  for  this  seenario  is 
approximately  $70,000  and  the  net  present  value  of  the  LTM  savings  is  approximately 
$1,500,000.  Thus,  the  net  savings  is  over  $1.4  million  over  10  years.  This  is  a  very  substantial 
net  benefit. 

As  stated  earlier,  many  of  the  parameters  used  in  these  seenarios  will  vary  from  site  to  site.  The 
eost  analysis  approaeh  summarized  in  Table  7-2  and  Table  7-3  ean  be  applied  to  any  sueh  set  of 
parameters.  This  simple  spreadsheet  approaeh  ean  be  used  to  sereen  sites  for  potential  benefits 
that  might  be  realized  from  applying  the  LTMO  software.  For  instanee,  for  sites  with  few 
monitoring  loeations  and  infrequent  sampling,  the  potential  savings  will  be  limited.  However, 
the  eost-benefit  examples  provided  above  clearly  indicate  that  net  savings  of  millions  of  dollars 
are  possible  across  the  universe  of  DoD  sites. 
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8.0  IMPLEMENTATION  ISSUES 


Software  Availability  and  Documentation 


The  anticipated  end-users  for  Sampling  Optimizer  and  Data  Tracker  include  government 
personnel  and  support  contractors  managing  groundwater  monitoring  programs.  A  copy  of  the 
software  executable  and  user’s  guide  is  available  on  the  Summit  Sampling  Optimizer  website 
(http://www.samplingoptimizer.com/)  for  free  and  immediate  download  by  government 
employees  and  educational  users  (those  accessing  with  “.gov”,  “.mil”  and  “.edu”  extensions). 
Input  data  files  from  this  project  that  can  be  used  as  sample  data  have  also  been  included  on  the 
website.  This  website  will  also  be  linked  to  the  ESTOP  and  Federal  Remediation  Technology 
Roundtable  websites. 

The  Summit  website  will  provide  a  form  for  contractors  to  government  sites  to  fill  out  to  obtain  a 
license  file  and  download  link  for  the  software.  Contractors  will  be  required  to  provide  evidence 
that  the  software  will  be  used  at  a  government  site  (e.g.,  a  government  work  order  or  letter  from 
government  personnel)  and  the  license  will  limit  the  software  to  only  be  able  to  work  with  data 
from  that  site.  Also,  contractors  will  need  to  renew  the  software  license  annually  for  continuing 
use  of  the  software.  The  free  software  license  does  not  include  technical  support  or  training, 
which  can  be  purchased  separately  (further  information  is  available  on  the  Summit  website). 
Other  private  sector  users  will  be  able  to  purchase  a  commercial  license  to  the  software  as 
needed.  Note  that  this  procedure  is  similar  to  those  employed  for  other  software  packages  such 
as  RACER  and  GMS. 

Ease  of  Use 


The  software  was  found  to  be  easy  to  use,  based  on  the  application  of  the  software  by  a  mid-level 
analyst  at  GeoTrans  with  no  ETMO  experience.  This  was  true  for  a  mid-level  analyst  who 
received  training  on  the  use  of  the  software  (for  one  of  the  three  demonstration  sites),  as  well  as 
for  a  mid-level  analyst  who  did  not  receive  training  on  the  software  (for  two  of  the  three 
demonstration  sites). 

In  addition,  the  EPA  group  that  applied  the  SO  functions  of  the  software  (including  Model 
Builder)  outside  of  our  project  reported  that:  “The  user  interface  was  very  easy  to  use. .  .User’s 
manual  was  an  excellent  reference  for  set-up  and  execution,  and  it  contained  clear  directions  for 
navigating  dialog  boxes,  setting  parameters,  formatting  input  files,  etc. .  .It  took  only  few  hours  to 
get  comfortable  using  the  software  (import/export,  model  set  up,  running  the  program).  The 
user’s  manual  was  very  helpful  in  this  aspect.  It  took  a  few  days  to  fully  understand  the  method, 
the  effects  of  changes  in  parameter  values,  and  the  results.” 

Key  Limitations  of  the  Current  Software 


The  software  has  some  limitation  that  will  impact  the  use  of  the  software  by  end-users.  Key 
limitations  (which  have  already  been  discussed  in  previous  portions  of  this  report)  are  indicated 
below. 
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•  The  software  interpolates  spatially  but  does  not  perform  interpolations  in  time.  This 
impaets  the  traeking  of  mass  and/or  mass  flux  when  the  distribution  of  sampling  is  not 
consistent  from  event  to  event.  It  also  impacts  the  performance  of  spatiotemporal 
redundancy  analysis,  resulting  in  more  conservative  results  than  spatial  redundancy 
analysis  when  the  sampling  locations  are  not  consistent  from  event  to  event.  The 
software  would  be  improved  if  there  was  a  feature  to  optionally  fill  in  missing  values  via 
temporal  interpolation. 

•  In  DT,  the  plots  of  concentration  versus  time  do  not  use  different  symbols  to  differentiate 
between  the  “background  data”  and  the  “current  data”.  The  software  would  be  improved 
if  different  symbols  were  used. 

•  In  DT,  the  software  does  not  allow  specific  historical  values  to  be  imported  and  plotted 
on  graphs  but  not  used  for  calculation  of  the  prediction  limits.  If  some  historical  values 
are  considered  potentially  anomalous,  those  values  have  to  either  be  included  as 
background  data  (such  that  prediction  limits  are  impacted)  or  completely  ignored.  The 
software  would  be  improved  if  such  values  could  be  imported  with  a  flag  so  that  they  can 
be  included  on  concentration  versus  time  plots  (with  a  different  symbol)  but  not  used  to 
calculate  the  prediction  limits. 

•  The  DT  portion  of  the  software  does  a  very  good  job  of  identifying  unexpected  values, 
but  does  not  indicate  whether  the  concentration  trend  for  a  specific  COC  at  a  specific 
well  is  increasing,  stable,  or  decreasing.  The  software  would  be  improved  if  that 
functionality  was  added. 

•  The  software  does  not  include  data  consolidation  or  recognition  of  flags  (e.g.,  for  non- 
detect  values).  This  requires  the  user  to  consolidate  the  data  into  sampling  events  during 
preparation  of  the  SO  input  fdes,  and  to  assign  “graphing  values”  for  non-detects  during 
preparation  of  the  input  files  for  SO  and  DT.  The  software  could  be  improved  if  this  type 
of  functionality  was  included  within  the  software. 

•  Plume  visualization  for  both  Model  Builder  and  Optimizer  also  allows  users  to  change 
the  zoom  scale  and  color  scale.  The  color  scale  is  a  linear  scale  allowing  users  to  define 
the  minimum  and  maximum  concentrations  for  each  COC.  Then  the  software  can  plot 
the  plume  maps  in  color  based  on  the  minimum  and  maximum  concentrations  defined. 
However,  it  does  not  provide  an  option  for  a  logarithmic  scale,  thus,  for  sites  with  a  very 
big  range  in  concentration,  it  cannot  plot  both  high-end  concentrations  and  low-end 
concentrations  with  sufficient  detail  (though  multiple  plots  with  different  ranges  could  be 
made  independently). 

The  first  bullet  listed  above  has  the  most  profound  implications  for  future  use  of  the  software. 

For  instance,  since  there  will  generally  be  an  uneven  distribution  of  sampling  in  different  events, 
the  use  of  spatiotemporal  redundancy  evaluation  may  not  be  advisable  in  most  cases,  because  the 
results  will  be  more  conservative  than  those  obtained  using  spatial  redundancy  evaluation  (with 
respect  to  elimination  of  wells).  An  additional  consideration  is  that  spatiotemporal  redundancy 
evaluation  requires  far  greater  computation  time  than  spatial  redundancy  evaluation  (for  the 
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demonstration  sites,  it  required  days  for  spatiotemporal  simulations  versus  hours  for  spatial 
simulations).  Thus,  a  prudent  approaeh  to  applying  the  software  for  reduetion  of  redundancy 
might  be  as  follows: 

•  Perform  spatial  optimization  rather  than  spatiotemporal  optimization 

•  Determine  if  eliminated  well  locations  in  one  or  more  of  the  recommended  plans  are 
reasonable  and  acceptable 

•  Qualitatively  specify  a  sampling  frequency  for  remaining  locations,  based  on  where 
changes  in  concentration  are  expected  and/or  are  of  greatest  concern 

•  Develop  rules  for  estimating  the  values  at  locations  not  sampled  in  a  specific  event  for 
developing  plume  maps  and/or  for  performing  mass  calculations  (e.g.,  latest  value, 
moving  average  of  latest  values,  etc.) 

This  approach  allows  the  user  to  utilize  the  most  powerful  and  beneficial  aspect  of  the  software, 
which  is  the  application  of  mathematical  optimization  in  conjunction  with  multiple  objectives  to 
develop  a  tradeoff  curve  for  evaluating  spatial  redundancy. 

Resulatory  Issues 


Regulatory  approval  regarding  the  implementation  of  LTMO  results  provided  by  the  software 
primarily  pertains  to  the  results  of  redundancy  evaluation  (i.e.,  the  SO  results).  Interaction  with 
regulators  regarding  implementation  of  results  at  the  three  demonstration  sites  was  not  a  specific 
part  of  this  ESTCP  project.  Site  personnel  for  the  demonstration  sites  indicated  that  the  types  or 
tradeoff  curves  produced  by  the  software  for  evaluating  redundancy  (based  on  mathematical 
optimization),  in  conjunction  with  the  comparison  of  plume  visualizations  with  and  without 
redundant  data  that  are  produced  by  the  software,  would  be  expected  to  be  convincing. 

Obtaining  regulatory  acceptance  of  the  software  will  require  two  major  steps:  1)  increasing 
awareness  of  LTMO  in  general,  and  awareness  of  this  software  in  particular,  within  the 
regulatory  community;  and  2)  making  site-specific  requests  to  regulators  for  modifying  an  LTM 
program  based  on  results  of  the  software.  The  project  team  has  offered  to  assist  each  of  the 
demonstration  sites  with  regulatory  issues  associated  with  LTMO,  but  no  such  assistance  has 
been  requested  to  date.  Lor  example,  the  site  team  at  the  GALB  site  indicated  they  would  like  to 
perform  further  analysis  on  their  own,  using  the  software,  before  presenting  results  to  regulators 
in  the  form  of  a  revised  LTM  plan.  Also,  given  the  long  schedule  of  our  project  and  the  fact  that 
the  most  recent  data  at  each  site  were  reserved  for  validation  in  our  project,  the  sites  would  be 
advised  to  repeat  the  analyses  using  up-to-date  data  before  incorporating  the  results  into  an  LTM 
program  revision  proposal.  Obtaining  ‘regulatory  acceptance”  of  the  software  will  ultimately 
require  that  LTM  modifications  based  on  software  recommendations  be  brought  before  site- 
specific  regulators. 
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